0.29.1原版

This commit is contained in:
anian
2026-07-02 19:14:14 +08:00
commit d94008f0fb
947 changed files with 174905 additions and 0 deletions
+304
View File
@@ -0,0 +1,304 @@
# Fileserver Package
## Overview
The `fileserver` package handles all binary file serving for Memos using native HTTP handlers. It was created to replace gRPC-based binary serving, which had limitations with HTTP range requests (required for Safari video/audio playback).
## Responsibilities
- Serve attachment binary files (images, videos, audio, documents)
- Serve user avatar images
- Handle HTTP range requests for video/audio streaming
- Authenticate requests using JWT tokens or Personal Access Tokens
- Check permissions for private content
- Generate and serve image thumbnails
- Prevent XSS attacks on uploaded content
- Support S3 external storage
## Architecture
### Design Principles
1. **Separation of Concerns**: Binary files via HTTP, metadata via gRPC
2. **DRY**: Imports auth constants from `api/v1` package (single source of truth)
3. **Security First**: Authentication, authorization, and XSS prevention
4. **Performance**: Native HTTP streaming with proper caching headers
### Package Structure
```
fileserver/
├── fileserver.go # Main service and HTTP handlers
├── README.md # This file
└── fileserver_test.go # Tests (to be added)
```
## API Endpoints
### 1. Attachment Binary
```
GET /file/attachments/:uid/:filename[?thumbnail=true]
```
**Parameters:**
- `uid` - Attachment unique identifier
- `filename` - Original filename
- `thumbnail` (optional) - Return thumbnail for images
**Authentication:** Required for non-public memos
**Response:**
- `200 OK` - File content with proper Content-Type
- `206 Partial Content` - For range requests (video/audio)
- `401 Unauthorized` - Authentication required
- `403 Forbidden` - User not authorized
- `404 Not Found` - Attachment not found
**Headers:**
- `Content-Type` - MIME type of the file
- `Cache-Control: public, max-age=3600`
- `Accept-Ranges: bytes` - For video/audio
- `Content-Range` - For partial responses (206)
### 2. User Avatar
```
GET /file/users/:identifier/avatar
```
**Parameters:**
- `identifier` - User ID (e.g., `1`) or username (e.g., `steven`)
**Authentication:** Not required (avatars are public)
**Response:**
- `200 OK` - Avatar image (PNG/JPEG)
- `404 Not Found` - User not found or no avatar set
**Headers:**
- `Content-Type` - image/png or image/jpeg
- `Cache-Control: public, max-age=3600`
## Authentication
### Supported Methods
The fileserver supports the following authentication methods:
1. **JWT Access Token** (`Authorization: Bearer {token}`)
- Short-lived tokens (15 minutes) for API access
- Stateless validation using JWT signature
- Extracts user ID from token claims
2. **Personal Access Token (PAT)** (`Authorization: Bearer {pat}`)
- Long-lived tokens for programmatic access
- Validates against database for revocation
- Prefixed with specific identifier
### Authentication Flow
```
Request → getCurrentUser()
├─→ Try Session Cookie
│ ├─→ Parse cookie value
│ ├─→ Get user from DB
│ ├─→ Validate session
│ └─→ Return user (if valid)
└─→ Try JWT Token
├─→ Parse Authorization header
├─→ Verify JWT signature
├─→ Get user from DB
├─→ Validate token in access tokens list
└─→ Return user (if valid)
```
### Permission Model
**Attachments:**
- Unlinked: Public (no auth required)
- Public memo: Public (no auth required)
- Protected memo: Requires authentication
- Private memo: Creator only
**Avatars:**
- Always public (no auth required)
## Key Functions
### HTTP Handlers
#### `serveAttachmentFile(c echo.Context) error`
Main handler for attachment binary serving.
**Flow:**
1. Extract UID from URL parameter
2. Fetch attachment from database
3. Check permissions (memo visibility)
4. Get binary blob (local file, S3, or database)
5. Handle thumbnail request (if applicable)
6. Set security headers (XSS prevention)
7. Serve with range request support (video/audio)
#### `serveUserAvatar(c echo.Context) error`
Main handler for user avatar serving.
**Flow:**
1. Extract identifier (ID or username) from URL
2. Lookup user in database
3. Check if avatar exists
4. Decode base64 data URI
5. Serve with proper content type and caching
### Authentication
#### `getCurrentUser(ctx, c) (*store.User, error)`
Authenticates request using session cookie or JWT token.
#### `authenticateBySession(ctx, cookie) (*store.User, error)`
Validates session cookie and returns authenticated user.
#### `authenticateByJWT(ctx, token) (*store.User, error)`
Validates JWT access token and returns authenticated user.
### Permission Checks
#### `checkAttachmentPermission(ctx, c, attachment) error`
Validates user has permission to access attachment based on memo visibility.
### File Operations
#### `getAttachmentBlob(attachment) ([]byte, error)`
Retrieves binary content from local storage, S3, or database.
#### `getOrGenerateThumbnail(ctx, attachment) ([]byte, error)`
Returns cached thumbnail or generates new one (with semaphore limiting).
### Utilities
#### `getUserByIdentifier(ctx, identifier) (*store.User, error)`
Finds user by ID (int) or username (string).
#### `extractImageInfo(dataURI) (type, base64, error)`
Parses data URI to extract MIME type and base64 data.
## Dependencies
### External Packages
- `github.com/labstack/echo/v5` - HTTP router and middleware
- `github.com/golang-jwt/jwt/v5` - JWT parsing and validation
- `github.com/disintegration/imaging` - Image thumbnail generation
- `golang.org/x/sync/semaphore` - Concurrency control for thumbnails
### Internal Packages
- `server/auth` - Authentication utilities
- `store` - Database operations
- `internal/profile` - Server configuration
- `internal/storage/s3` - S3 storage client
## Configuration
### Constants
Auth-related constants are imported from `server/auth`:
- `auth.RefreshTokenCookieName` - "memos_refresh"
- `auth.PersonalAccessTokenPrefix` - PAT identifier prefix
Package-specific constants:
- `ThumbnailCacheFolder` - ".thumbnail_cache"
- `thumbnailMaxSize` - 600px
- `SupportedThumbnailMimeTypes` - ["image/png", "image/jpeg"]
## Error Handling
All handlers return Echo HTTP errors with appropriate status codes:
```go
// Bad request
echo.NewHTTPError(http.StatusBadRequest, "message")
// Unauthorized (no auth)
echo.NewHTTPError(http.StatusUnauthorized, "message")
// Forbidden (auth but no permission)
echo.NewHTTPError(http.StatusForbidden, "message")
// Not found
echo.NewHTTPError(http.StatusNotFound, "message")
// Internal error
echo.NewHTTPError(http.StatusInternalServerError, "message").SetInternal(err)
```
## Security Considerations
### 1. XSS Prevention
SVG and HTML files are served as `application/octet-stream` to prevent script execution:
```go
if contentType == "image/svg+xml" ||
contentType == "text/html" ||
contentType == "application/xhtml+xml" {
contentType = "application/octet-stream"
}
```
### 2. Authentication
Private content requires valid JWT access token or Personal Access Token.
### 3. Authorization
Memo visibility rules enforced before serving attachments.
### 4. Input Validation
- Attachment UID validated from database
- User identifier validated (ID or username)
- Range requests validated before processing
## Performance Optimizations
### 1. Thumbnail Caching
Thumbnails cached on disk to avoid regeneration:
- Cache location: `{data_dir}/.thumbnail_cache/`
- Filename: `{attachment_id}{extension}`
- Semaphore limits concurrent generation (max 3)
### 2. HTTP Range Requests
Video/audio files use `http.ServeContent()` for efficient streaming:
- Automatic range parsing
- Efficient memory usage (streaming, not loading full file)
- Safari-compatible partial content responses
### 3. Caching Headers
All responses include cache headers:
```
Cache-Control: public, max-age=3600
```
### 4. S3 External Links
S3 files served via presigned URLs (no server download).
## Testing
### Unit Tests (To Add)
See SAFARI_FIX.md for recommended test coverage.
### Manual Testing
```bash
# Test attachment
curl "http://localhost:8081/file/attachments/{uid}/file.jpg"
# Test avatar by username
curl "http://localhost:8081/file/users/steven/avatar"
# Test range request
curl -H "Range: bytes=0-999" "http://localhost:8081/file/attachments/{uid}/video.mp4"
```
## Future Improvements
See SAFARI_FIX.md section "Future Improvements" for planned enhancements.
## Related Documentation
- [SAFARI_FIX.md](../../../SAFARI_FIX.md) - Full migration guide
- [server/router/api/v1/auth.go](../api/v1/auth.go) - Auth constants source of truth
- [RFC 7233](https://tools.ietf.org/html/rfc7233) - HTTP Range Requests spec
+764
View File
@@ -0,0 +1,764 @@
package fileserver
import (
"bytes"
"context"
"encoding/base64"
"fmt"
"io"
"log/slog"
"net/http"
"os"
"path/filepath"
"regexp"
"strings"
"time"
"github.com/disintegration/imaging"
"github.com/labstack/echo/v5"
"github.com/pkg/errors"
"golang.org/x/sync/semaphore"
"github.com/usememos/memos/internal/motionphoto"
"github.com/usememos/memos/internal/profile"
"github.com/usememos/memos/internal/storage/s3"
storepb "github.com/usememos/memos/proto/gen/store"
"github.com/usememos/memos/server/auth"
"github.com/usememos/memos/store"
)
// Constants for file serving configuration.
const (
// ThumbnailCacheFolder is the folder name where thumbnail images are stored.
ThumbnailCacheFolder = ".thumbnail_cache"
// MotionCacheFolder is the folder name where extracted motion clips are stored.
MotionCacheFolder = ".motion_cache"
// thumbnailMaxSize is the maximum dimension (width or height) for thumbnails.
thumbnailMaxSize = 600
// thumbnailMetadataProbeSize is the maximum number of original image bytes inspected
// before thumbnail generation to detect metadata that the JPEG thumbnail pipeline cannot preserve.
thumbnailMetadataProbeSize = 1 << 20
// maxConcurrentThumbnails limits concurrent thumbnail generation to prevent memory exhaustion.
maxConcurrentThumbnails = 3
// cacheMaxAge is the max-age value for Cache-Control headers (1 hour).
cacheMaxAge = "public, max-age=3600"
)
// xssUnsafeTypes contains MIME types that could execute scripts if served directly.
// These are served as application/octet-stream to prevent XSS attacks.
var xssUnsafeTypes = map[string]bool{
"text/html": true,
"text/javascript": true,
"application/javascript": true,
"application/x-javascript": true,
"text/xml": true,
"application/xml": true,
"application/xhtml+xml": true,
}
// thumbnailSupportedTypes contains image MIME types that support thumbnail generation.
var thumbnailSupportedTypes = map[string]bool{
"image/png": true,
"image/jpeg": true,
"image/jpg": true,
"image/heic": true,
"image/heif": true,
"image/webp": true,
}
// avatarAllowedTypes contains MIME types allowed for user avatars.
var avatarAllowedTypes = map[string]bool{
"image/png": true,
"image/jpeg": true,
"image/jpg": true,
"image/gif": true,
"image/webp": true,
"image/heic": true,
"image/heif": true,
}
// SupportedThumbnailMimeTypes is the exported list of thumbnail-supported MIME types.
var SupportedThumbnailMimeTypes = []string{
"image/png",
"image/jpeg",
"image/jpg",
"image/heic",
"image/heif",
"image/webp",
}
var errUseOriginalForThumbnail = errors.New("serve original image instead of metadata-stripping thumbnail")
// dataURIRegex parses data URI format: data:image/png;base64,iVBORw0KGgo...
var dataURIRegex = regexp.MustCompile(`^data:(?P<type>[^;]+);base64,(?P<base64>.+)`)
// FileServerService handles HTTP file serving with proper range request support.
type FileServerService struct {
Profile *profile.Profile
Store *store.Store
authenticator *auth.Authenticator
// thumbnailSemaphore limits concurrent thumbnail generation.
thumbnailSemaphore *semaphore.Weighted
}
// NewFileServerService creates a new file server service.
func NewFileServerService(profile *profile.Profile, store *store.Store, secret string) *FileServerService {
return &FileServerService{
Profile: profile,
Store: store,
authenticator: auth.NewAuthenticator(store, secret),
thumbnailSemaphore: semaphore.NewWeighted(maxConcurrentThumbnails),
}
}
// RegisterRoutes registers HTTP file serving routes.
func (s *FileServerService) RegisterRoutes(echoServer *echo.Echo) {
fileGroup := echoServer.Group("/file")
fileGroup.GET("/attachments/:uid/:filename", s.serveAttachmentFile)
fileGroup.GET("/users/:identifier/avatar", s.serveUserAvatar)
}
// =============================================================================
// HTTP Handlers
// =============================================================================
// serveAttachmentFile serves attachment binary content using native HTTP.
func (s *FileServerService) serveAttachmentFile(c *echo.Context) error {
ctx := c.Request().Context()
uid := c.Param("uid")
wantThumbnail := c.QueryParam("thumbnail") == "true"
wantMotion := c.QueryParam("motion") == "true"
attachment, err := s.Store.GetAttachment(ctx, &store.FindAttachment{
UID: &uid,
GetBlob: true,
})
if err != nil {
return echo.NewHTTPError(http.StatusInternalServerError, "failed to get attachment").Wrap(err)
}
if attachment == nil {
return echo.NewHTTPError(http.StatusNotFound, "attachment not found")
}
if err := s.checkAttachmentPermission(ctx, c, attachment); err != nil {
return err
}
if wantMotion {
return s.serveMotionClip(c, attachment)
}
contentType := s.sanitizeContentType(attachment.Type)
// Stream video/audio to avoid loading entire file into memory.
if isMediaType(attachment.Type) {
return s.serveMediaStream(c, attachment, contentType)
}
return s.serveStaticFile(c, attachment, contentType, wantThumbnail)
}
// serveUserAvatar serves user avatar images.
func (s *FileServerService) serveUserAvatar(c *echo.Context) error {
ctx := c.Request().Context()
identifier := c.Param("identifier")
user, err := s.getUserByUsername(ctx, identifier)
if err != nil {
return echo.NewHTTPError(http.StatusInternalServerError, "failed to get user").Wrap(err)
}
if user == nil {
return echo.NewHTTPError(http.StatusNotFound, "user not found")
}
if user.AvatarURL == "" {
return echo.NewHTTPError(http.StatusNotFound, "avatar not found")
}
imageType, imageData, err := s.parseDataURI(user.AvatarURL)
if err != nil {
return echo.NewHTTPError(http.StatusInternalServerError, "failed to parse avatar data").Wrap(err)
}
if !avatarAllowedTypes[imageType] {
return echo.NewHTTPError(http.StatusBadRequest, "invalid avatar image type")
}
setSecurityHeaders(c)
c.Response().Header().Set(echo.HeaderContentType, imageType)
c.Response().Header().Set(echo.HeaderCacheControl, cacheMaxAge)
return c.Blob(http.StatusOK, imageType, imageData)
}
// =============================================================================
// File Serving Methods
// =============================================================================
// serveMediaStream serves video/audio files using streaming to avoid memory exhaustion.
func (s *FileServerService) serveMediaStream(c *echo.Context, attachment *store.Attachment, contentType string) error {
setSecurityHeaders(c)
setMediaHeaders(c, contentType, attachment.Type)
switch attachment.StorageType {
case storepb.AttachmentStorageType_LOCAL:
filePath, err := s.resolveLocalPath(attachment.Reference)
if err != nil {
return echo.NewHTTPError(http.StatusInternalServerError, "failed to resolve file path").Wrap(err)
}
http.ServeFile(c.Response(), c.Request(), filePath)
return nil
case storepb.AttachmentStorageType_S3:
presignURL, err := s.getS3PresignedURL(c.Request().Context(), attachment)
if err != nil {
return echo.NewHTTPError(http.StatusInternalServerError, "failed to generate presigned URL").Wrap(err)
}
return c.Redirect(http.StatusTemporaryRedirect, presignURL)
default:
// Database storage fallback.
modTime := time.Unix(attachment.UpdatedTs, 0)
http.ServeContent(c.Response(), c.Request(), attachment.Filename, modTime, bytes.NewReader(attachment.Blob))
return nil
}
}
// serveStaticFile serves non-streaming files (images, documents, etc.).
func (s *FileServerService) serveStaticFile(c *echo.Context, attachment *store.Attachment, contentType string, wantThumbnail bool) error {
// Generate thumbnail for supported image types.
if wantThumbnail && thumbnailSupportedTypes[attachment.Type] {
if thumbnailBlob, err := s.getOrGenerateThumbnail(c.Request().Context(), attachment); err != nil {
if !errors.Is(err, errUseOriginalForThumbnail) {
slog.Warn("failed to get thumbnail", "error", err)
}
} else {
setSecurityHeaders(c)
setMediaHeaders(c, "image/jpeg", attachment.Type)
return c.Blob(http.StatusOK, "image/jpeg", thumbnailBlob)
}
}
setSecurityHeaders(c)
setMediaHeaders(c, contentType, attachment.Type)
// Force download for non-media files to prevent XSS execution.
if !strings.HasPrefix(contentType, "image/") && contentType != "application/pdf" {
c.Response().Header().Set(echo.HeaderContentDisposition, fmt.Sprintf("attachment; filename=%q", attachment.Filename))
}
switch attachment.StorageType {
case storepb.AttachmentStorageType_LOCAL:
filePath, err := s.resolveLocalPath(attachment.Reference)
if err != nil {
return echo.NewHTTPError(http.StatusInternalServerError, "failed to resolve file path").Wrap(err)
}
http.ServeFile(c.Response(), c.Request(), filePath)
return nil
case storepb.AttachmentStorageType_S3:
reader, err := s.getAttachmentReader(c.Request().Context(), attachment)
if err != nil {
return echo.NewHTTPError(http.StatusInternalServerError, "failed to get attachment reader").Wrap(err)
}
defer reader.Close()
return c.Stream(http.StatusOK, contentType, reader)
default:
return c.Blob(http.StatusOK, contentType, attachment.Blob)
}
}
// =============================================================================
// Storage Operations
// =============================================================================
// getAttachmentBlob retrieves the binary content of an attachment from storage.
func (s *FileServerService) getAttachmentBlob(attachment *store.Attachment) ([]byte, error) {
switch attachment.StorageType {
case storepb.AttachmentStorageType_LOCAL:
return s.readLocalFile(attachment.Reference)
case storepb.AttachmentStorageType_S3:
return s.downloadFromS3(context.Background(), attachment)
default:
return attachment.Blob, nil
}
}
// getAttachmentReader returns a reader for streaming attachment content.
func (s *FileServerService) getAttachmentReader(ctx context.Context, attachment *store.Attachment) (io.ReadCloser, error) {
switch attachment.StorageType {
case storepb.AttachmentStorageType_LOCAL:
filePath, err := s.resolveLocalPath(attachment.Reference)
if err != nil {
return nil, err
}
file, err := os.Open(filePath)
if err != nil {
if os.IsNotExist(err) {
return nil, errors.Wrap(err, "file not found")
}
return nil, errors.Wrap(err, "failed to open file")
}
return file, nil
case storepb.AttachmentStorageType_S3:
s3Client, s3Object, err := s.createS3Client(attachment)
if err != nil {
return nil, err
}
reader, err := s3Client.GetObjectStream(ctx, s3Object.Key)
if err != nil {
return nil, errors.Wrap(err, "failed to stream from S3")
}
return reader, nil
default:
return io.NopCloser(bytes.NewReader(attachment.Blob)), nil
}
}
// resolveLocalPath converts a storage reference to an absolute file path.
func (s *FileServerService) resolveLocalPath(reference string) (string, error) {
filePath := filepath.FromSlash(reference)
if !filepath.IsAbs(filePath) {
filePath = filepath.Join(s.Profile.Data, filePath)
}
return filePath, nil
}
// readLocalFile reads the entire contents of a local file.
func (s *FileServerService) readLocalFile(reference string) ([]byte, error) {
filePath, err := s.resolveLocalPath(reference)
if err != nil {
return nil, err
}
file, err := os.Open(filePath)
if err != nil {
if os.IsNotExist(err) {
return nil, errors.Wrap(err, "file not found")
}
return nil, errors.Wrap(err, "failed to open file")
}
defer file.Close()
blob, err := io.ReadAll(file)
if err != nil {
return nil, errors.Wrap(err, "failed to read file")
}
return blob, nil
}
// createS3Client creates an S3 client from attachment payload.
func (*FileServerService) createS3Client(attachment *store.Attachment) (*s3.Client, *storepb.AttachmentPayload_S3Object, error) {
if attachment.Payload == nil {
return nil, nil, errors.New("attachment payload is missing")
}
s3Object := attachment.Payload.GetS3Object()
if s3Object == nil {
return nil, nil, errors.New("S3 object payload is missing")
}
if s3Object.S3Config == nil {
return nil, nil, errors.New("S3 config is missing")
}
if s3Object.Key == "" {
return nil, nil, errors.New("S3 object key is missing")
}
client, err := s3.NewClient(context.Background(), s3Object.S3Config)
if err != nil {
return nil, nil, errors.Wrap(err, "failed to create S3 client")
}
return client, s3Object, nil
}
// downloadFromS3 downloads the entire object from S3.
func (s *FileServerService) downloadFromS3(ctx context.Context, attachment *store.Attachment) ([]byte, error) {
client, s3Object, err := s.createS3Client(attachment)
if err != nil {
return nil, err
}
blob, err := client.GetObject(ctx, s3Object.Key)
if err != nil {
return nil, errors.Wrap(err, "failed to download from S3")
}
return blob, nil
}
// getS3PresignedURL generates a presigned URL for direct S3 access.
func (s *FileServerService) getS3PresignedURL(ctx context.Context, attachment *store.Attachment) (string, error) {
client, s3Object, err := s.createS3Client(attachment)
if err != nil {
return "", err
}
url, err := client.PresignGetObject(ctx, s3Object.Key)
if err != nil {
return "", errors.Wrap(err, "failed to presign URL")
}
return url, nil
}
// =============================================================================
// Thumbnail Generation
// =============================================================================
// getOrGenerateThumbnail returns the thumbnail image of the attachment.
// Uses semaphore to limit concurrent thumbnail generation and prevent memory exhaustion.
func (s *FileServerService) getOrGenerateThumbnail(ctx context.Context, attachment *store.Attachment) ([]byte, error) {
thumbnailPath, err := s.getThumbnailPath(attachment)
if err != nil {
return nil, err
}
// Fast path: return cached thumbnail if exists.
if blob, err := s.readCachedThumbnail(thumbnailPath); err == nil {
return blob, nil
}
useOriginal, err := s.shouldUseOriginalForThumbnail(ctx, attachment)
if err != nil {
return nil, err
}
if useOriginal {
return nil, errUseOriginalForThumbnail
}
// Acquire semaphore to limit concurrent generation.
if err := s.thumbnailSemaphore.Acquire(ctx, 1); err != nil {
return nil, errors.Wrap(err, "failed to acquire semaphore")
}
defer s.thumbnailSemaphore.Release(1)
// Double-check after acquiring semaphore (another goroutine may have generated it).
if blob, err := s.readCachedThumbnail(thumbnailPath); err == nil {
return blob, nil
}
return s.generateThumbnail(ctx, attachment, thumbnailPath)
}
// getThumbnailPath returns the file path for a cached thumbnail.
func (s *FileServerService) getThumbnailPath(attachment *store.Attachment) (string, error) {
cacheFolder := filepath.Join(s.Profile.Data, ThumbnailCacheFolder)
if err := os.MkdirAll(cacheFolder, os.ModePerm); err != nil {
return "", errors.Wrap(err, "failed to create thumbnail cache folder")
}
filename := fmt.Sprintf("%s.v2.jpeg", attachment.UID)
return filepath.Join(cacheFolder, filename), nil
}
func (s *FileServerService) shouldUseOriginalForThumbnail(ctx context.Context, attachment *store.Attachment) (bool, error) {
if attachment.Type == "image/heic" || attachment.Type == "image/heif" {
return true, nil
}
if attachment.Type != "image/jpeg" && attachment.Type != "image/jpg" && attachment.Type != "image/png" && attachment.Type != "image/webp" {
return false, nil
}
reader, err := s.getAttachmentReader(ctx, attachment)
if err != nil {
return false, errors.Wrap(err, "failed to open image for metadata probe")
}
defer reader.Close()
probe, err := io.ReadAll(io.LimitReader(reader, thumbnailMetadataProbeSize))
if err != nil {
return false, errors.Wrap(err, "failed to read image metadata probe")
}
return hasThumbnailSensitiveMetadata(attachment.Type, probe), nil
}
func hasThumbnailSensitiveMetadata(mimeType string, data []byte) bool {
if mimeType == "image/heic" || mimeType == "image/heif" {
return true
}
for _, marker := range [][]byte{
[]byte("ICC_PROFILE"),
[]byte("iCCP"),
[]byte("ICCP"),
[]byte("cICP"),
[]byte("mDCv"),
[]byte("cLLi"),
} {
if bytes.Contains(data, marker) {
return true
}
}
lowerData := strings.ToLower(string(data))
for _, marker := range []string{
"hdrgm:",
"hdr gain map",
"hdrgainmap",
"gainmap",
"ultrahdr",
"adobe:hdrgainmap",
"aux:hdr",
"auxiliaryimagetype",
"display p3",
"display-p3",
"rec.2020",
"bt.2020",
"arib-std-b67",
"smpte st 2084",
} {
if strings.Contains(lowerData, marker) {
return true
}
}
return false
}
// readCachedThumbnail reads a thumbnail from the cache directory.
func (*FileServerService) readCachedThumbnail(path string) ([]byte, error) {
file, err := os.Open(path)
if err != nil {
return nil, err
}
defer file.Close()
return io.ReadAll(file)
}
// generateThumbnail creates a new thumbnail and saves it to disk.
func (s *FileServerService) generateThumbnail(ctx context.Context, attachment *store.Attachment, thumbnailPath string) ([]byte, error) {
reader, err := s.getAttachmentReader(ctx, attachment)
if err != nil {
return nil, errors.Wrap(err, "failed to get attachment reader")
}
defer reader.Close()
img, err := imaging.Decode(reader, imaging.AutoOrientation(true))
if err != nil {
return nil, errors.Wrap(err, "failed to decode image")
}
width, height := img.Bounds().Dx(), img.Bounds().Dy()
thumbnailWidth, thumbnailHeight := calculateThumbnailDimensions(width, height)
thumbnailImage := imaging.Resize(img, thumbnailWidth, thumbnailHeight, imaging.Lanczos)
output, err := os.Create(thumbnailPath)
if err != nil {
return nil, errors.Wrap(err, "failed to create thumbnail file")
}
defer output.Close()
if err := imaging.Encode(output, thumbnailImage, imaging.JPEG, imaging.JPEGQuality(90)); err != nil {
return nil, errors.Wrap(err, "failed to save thumbnail")
}
return s.readCachedThumbnail(thumbnailPath)
}
// calculateThumbnailDimensions calculates the target dimensions for a thumbnail.
// The largest dimension is constrained to thumbnailMaxSize while maintaining aspect ratio.
// Small images are not enlarged.
func calculateThumbnailDimensions(width, height int) (int, int) {
if max(width, height) <= thumbnailMaxSize {
return width, height
}
if width >= height {
return thumbnailMaxSize, 0 // Landscape: constrain width.
}
return 0, thumbnailMaxSize // Portrait: constrain height.
}
func (s *FileServerService) serveMotionClip(c *echo.Context, attachment *store.Attachment) error {
motionMedia := attachment.Payload.GetMotionMedia()
if motionMedia == nil || motionMedia.Family != storepb.MotionMediaFamily_ANDROID_MOTION_PHOTO || !motionMedia.HasEmbeddedVideo {
return echo.NewHTTPError(http.StatusBadRequest, "attachment does not have motion clip")
}
clipBlob, err := s.getOrExtractMotionClip(c.Request().Context(), attachment)
if err != nil {
return echo.NewHTTPError(http.StatusInternalServerError, "failed to get motion clip").Wrap(err)
}
setSecurityHeaders(c)
setMediaHeaders(c, "video/mp4", "video/mp4")
modTime := time.Unix(attachment.UpdatedTs, 0)
http.ServeContent(c.Response(), c.Request(), attachment.UID+".mp4", modTime, bytes.NewReader(clipBlob))
return nil
}
func (s *FileServerService) getOrExtractMotionClip(_ context.Context, attachment *store.Attachment) ([]byte, error) {
motionPath, err := s.getMotionPath(attachment)
if err != nil {
return nil, err
}
if blob, err := s.readCachedThumbnail(motionPath); err == nil {
return blob, nil
}
blob, err := s.getAttachmentBlob(attachment)
if err != nil {
return nil, err
}
videoBlob, _ := motionphoto.ExtractVideo(blob)
if len(videoBlob) == 0 {
return nil, errors.New("motion video not found")
}
if err := os.WriteFile(motionPath, videoBlob, 0644); err != nil {
return nil, errors.Wrap(err, "failed to cache motion clip")
}
return videoBlob, nil
}
func (s *FileServerService) getMotionPath(attachment *store.Attachment) (string, error) {
cacheFolder := filepath.Join(s.Profile.Data, MotionCacheFolder)
if err := os.MkdirAll(cacheFolder, os.ModePerm); err != nil {
return "", errors.Wrap(err, "failed to create motion cache folder")
}
return filepath.Join(cacheFolder, attachment.UID+".mp4"), nil
}
// =============================================================================
// Authentication & Authorization
// =============================================================================
// checkAttachmentPermission verifies the user has permission to access the attachment.
func (s *FileServerService) checkAttachmentPermission(ctx context.Context, c *echo.Context, attachment *store.Attachment) error {
// For unlinked attachments, only the creator can access.
if attachment.MemoID == nil {
user, err := s.getCurrentUser(ctx, c)
if err != nil {
return echo.NewHTTPError(http.StatusInternalServerError, "failed to get current user").Wrap(err)
}
if user == nil {
return echo.NewHTTPError(http.StatusUnauthorized, "unauthorized access")
}
if user.ID != attachment.CreatorID && user.Role != store.RoleAdmin {
return echo.NewHTTPError(http.StatusForbidden, "forbidden access")
}
return nil
}
memo, err := s.Store.GetMemo(ctx, &store.FindMemo{ID: attachment.MemoID})
if err != nil {
return echo.NewHTTPError(http.StatusInternalServerError, "failed to find memo").Wrap(err)
}
if memo == nil {
return echo.NewHTTPError(http.StatusNotFound, "memo not found")
}
if memo.Visibility == store.Public {
return nil
}
// Check share token fallback: allow access if request carries a valid, non-expired share token
// that was issued for this specific memo. This covers attachment requests made from the shared
// memo page for private or protected memos.
if shareToken := (*c).QueryParam("share_token"); shareToken != "" {
ms, err := s.Store.GetMemoShare(ctx, &store.FindMemoShare{UID: &shareToken})
if err == nil && ms != nil && !isMemoShareExpired(ms) && ms.MemoID == memo.ID {
return nil
}
}
user, err := s.getCurrentUser(ctx, c)
if err != nil {
return echo.NewHTTPError(http.StatusInternalServerError, "failed to get current user").Wrap(err)
}
if user == nil {
return echo.NewHTTPError(http.StatusUnauthorized, "unauthorized access")
}
if memo.Visibility == store.Private && user.ID != memo.CreatorID && user.Role != store.RoleAdmin {
return echo.NewHTTPError(http.StatusForbidden, "forbidden access")
}
return nil
}
// getCurrentUser retrieves the current authenticated user from the request.
// Authentication priority: Bearer token (Access Token V2 or PAT) > Refresh token cookie.
func (s *FileServerService) getCurrentUser(ctx context.Context, c *echo.Context) (*store.User, error) {
authHeader := c.Request().Header.Get(echo.HeaderAuthorization)
cookieHeader := c.Request().Header.Get("Cookie")
return s.authenticator.AuthenticateToUser(ctx, authHeader, cookieHeader)
}
// getUserByUsername finds a user by username only.
func (s *FileServerService) getUserByUsername(ctx context.Context, identifier string) (*store.User, error) {
return s.Store.GetUser(ctx, &store.FindUser{Username: &identifier})
}
// =============================================================================
// Helper Functions
// =============================================================================
// sanitizeContentType converts potentially dangerous MIME types to safe alternatives.
func (*FileServerService) sanitizeContentType(mimeType string) string {
contentType := mimeType
if strings.HasPrefix(contentType, "text/") {
contentType += "; charset=utf-8"
}
// Normalize for case-insensitive lookup.
if xssUnsafeTypes[strings.ToLower(mimeType)] {
return "application/octet-stream"
}
return contentType
}
// parseDataURI extracts MIME type and decoded data from a data URI.
func (*FileServerService) parseDataURI(dataURI string) (string, []byte, error) {
matches := dataURIRegex.FindStringSubmatch(dataURI)
if len(matches) != 3 {
return "", nil, errors.New("invalid data URI format")
}
imageType := matches[1]
imageData, err := base64.StdEncoding.DecodeString(matches[2])
if err != nil {
return "", nil, errors.Wrap(err, "failed to decode base64 data")
}
return imageType, imageData, nil
}
// isMediaType checks if the MIME type is video or audio.
func isMediaType(mimeType string) bool {
return strings.HasPrefix(mimeType, "video/") || strings.HasPrefix(mimeType, "audio/")
}
// setSecurityHeaders sets common security headers for all responses.
func setSecurityHeaders(c *echo.Context) {
h := c.Response().Header()
h.Set("X-Content-Type-Options", "nosniff")
h.Set("X-Frame-Options", "DENY")
h.Set("Content-Security-Policy", "default-src 'none'; style-src 'unsafe-inline';")
}
// setMediaHeaders sets headers for media file responses.
func setMediaHeaders(c *echo.Context, contentType, originalType string) {
h := c.Response().Header()
h.Set(echo.HeaderContentType, contentType)
h.Set(echo.HeaderCacheControl, cacheMaxAge)
// Support HDR/wide color gamut for images and videos.
if strings.HasPrefix(originalType, "image/") || strings.HasPrefix(originalType, "video/") {
h.Set("Color-Gamut", "srgb, p3, rec2020")
}
}
// isMemoShareExpired returns true if the share has a defined expiry that has already passed.
func isMemoShareExpired(ms *store.MemoShare) bool {
return ms.ExpiresTs != nil && time.Now().Unix() > *ms.ExpiresTs
}
+435
View File
@@ -0,0 +1,435 @@
package fileserver
import (
"bytes"
"context"
"encoding/binary"
"fmt"
"hash/crc32"
"image"
"image/color"
"image/png"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/labstack/echo/v5"
"github.com/stretchr/testify/require"
"github.com/usememos/memos/internal/markdown"
"github.com/usememos/memos/internal/profile"
"github.com/usememos/memos/internal/testutil"
apiv1 "github.com/usememos/memos/proto/gen/api/v1"
"github.com/usememos/memos/server/auth"
apiv1service "github.com/usememos/memos/server/router/api/v1"
"github.com/usememos/memos/store"
teststore "github.com/usememos/memos/store/test"
)
func TestServeAttachmentFile_ShareTokenAllowsDirectMemoAttachment(t *testing.T) {
ctx := context.Background()
svc, fs, _, cleanup := newShareAttachmentTestServices(ctx, t)
defer cleanup()
creator, err := svc.Store.CreateUser(ctx, &store.User{
Username: "share-parent-owner",
Role: store.RoleUser,
Email: "share-parent-owner@example.com",
})
require.NoError(t, err)
creatorCtx := context.WithValue(ctx, auth.UserIDContextKey, creator.ID)
attachment, err := svc.CreateAttachment(creatorCtx, &apiv1.CreateAttachmentRequest{
Attachment: &apiv1.Attachment{
Filename: "memo.txt",
Type: "text/plain",
Content: []byte("memo attachment"),
},
})
require.NoError(t, err)
parentMemo, err := svc.CreateMemo(creatorCtx, &apiv1.CreateMemoRequest{
Memo: &apiv1.Memo{
Content: "shared parent",
Visibility: apiv1.Visibility_PROTECTED,
Attachments: []*apiv1.Attachment{
{Name: attachment.Name},
},
},
})
require.NoError(t, err)
share, err := svc.CreateMemoShare(creatorCtx, &apiv1.CreateMemoShareRequest{
Parent: parentMemo.Name,
MemoShare: &apiv1.MemoShare{},
})
require.NoError(t, err)
shareToken := share.Name[strings.LastIndex(share.Name, "/")+1:]
e := echo.New()
fs.RegisterRoutes(e)
req := httptest.NewRequest(http.MethodGet, fmt.Sprintf("/file/%s/%s?share_token=%s", attachment.Name, attachment.Filename, shareToken), nil)
rec := httptest.NewRecorder()
e.ServeHTTP(rec, req)
require.Equal(t, http.StatusOK, rec.Code)
require.Equal(t, "memo attachment", rec.Body.String())
}
func TestServeAttachmentFile_LocalStaticFileSupportsRangeRequests(t *testing.T) {
ctx := context.Background()
svc, fs, _, cleanup := newShareAttachmentTestServices(ctx, t)
defer cleanup()
creator, err := svc.Store.CreateUser(ctx, &store.User{
Username: "range-owner",
Role: store.RoleUser,
Email: "range-owner@example.com",
})
require.NoError(t, err)
creatorCtx := context.WithValue(ctx, auth.UserIDContextKey, creator.ID)
attachment, err := svc.CreateAttachment(creatorCtx, &apiv1.CreateAttachmentRequest{
Attachment: &apiv1.Attachment{
Filename: "range.txt",
Type: "text/plain",
Content: []byte("0123456789"),
},
})
require.NoError(t, err)
_, err = svc.CreateMemo(creatorCtx, &apiv1.CreateMemoRequest{
Memo: &apiv1.Memo{
Content: "range memo",
Visibility: apiv1.Visibility_PUBLIC,
Attachments: []*apiv1.Attachment{
{Name: attachment.Name},
},
},
})
require.NoError(t, err)
e := echo.New()
fs.RegisterRoutes(e)
req := httptest.NewRequest(http.MethodGet, fmt.Sprintf("/file/%s/%s", attachment.Name, attachment.Filename), nil)
req.Header.Set("Range", "bytes=2-5")
rec := httptest.NewRecorder()
e.ServeHTTP(rec, req)
require.Equal(t, http.StatusPartialContent, rec.Code)
require.Equal(t, "2345", rec.Body.String())
require.Equal(t, "bytes 2-5/10", rec.Header().Get("Content-Range"))
}
func TestServeAttachmentFile_ShareTokenRejectsCommentAttachment(t *testing.T) {
ctx := context.Background()
svc, fs, _, cleanup := newShareAttachmentTestServices(ctx, t)
defer cleanup()
creator, err := svc.Store.CreateUser(ctx, &store.User{
Username: "private-parent-owner",
Role: store.RoleUser,
Email: "private-parent-owner@example.com",
})
require.NoError(t, err)
creatorCtx := context.WithValue(ctx, auth.UserIDContextKey, creator.ID)
commenter, err := svc.Store.CreateUser(ctx, &store.User{
Username: "share-commenter",
Role: store.RoleUser,
Email: "share-commenter@example.com",
})
require.NoError(t, err)
commenterCtx := context.WithValue(ctx, auth.UserIDContextKey, commenter.ID)
parentMemo, err := svc.CreateMemo(creatorCtx, &apiv1.CreateMemoRequest{
Memo: &apiv1.Memo{
Content: "shared parent",
Visibility: apiv1.Visibility_PROTECTED,
},
})
require.NoError(t, err)
commentAttachment, err := svc.CreateAttachment(commenterCtx, &apiv1.CreateAttachmentRequest{
Attachment: &apiv1.Attachment{
Filename: "comment.txt",
Type: "text/plain",
Content: []byte("comment attachment"),
},
})
require.NoError(t, err)
_, err = svc.CreateMemoComment(commenterCtx, &apiv1.CreateMemoCommentRequest{
Name: parentMemo.Name,
Comment: &apiv1.Memo{
Content: "comment with attachment",
Visibility: apiv1.Visibility_PROTECTED,
Attachments: []*apiv1.Attachment{
{Name: commentAttachment.Name},
},
},
})
require.NoError(t, err)
share, err := svc.CreateMemoShare(creatorCtx, &apiv1.CreateMemoShareRequest{
Parent: parentMemo.Name,
MemoShare: &apiv1.MemoShare{},
})
require.NoError(t, err)
shareToken := share.Name[strings.LastIndex(share.Name, "/")+1:]
e := echo.New()
fs.RegisterRoutes(e)
req := httptest.NewRequest(http.MethodGet, fmt.Sprintf("/file/%s/%s?share_token=%s", commentAttachment.Name, commentAttachment.Filename, shareToken), nil)
rec := httptest.NewRecorder()
e.ServeHTTP(rec, req)
require.Equal(t, http.StatusUnauthorized, rec.Code)
}
func TestServeAttachmentFile_MotionClip(t *testing.T) {
ctx := context.Background()
svc, fs, _, cleanup := newShareAttachmentTestServices(ctx, t)
defer cleanup()
creator, err := svc.Store.CreateUser(ctx, &store.User{
Username: "motion-owner",
Role: store.RoleUser,
Email: "motion-owner@example.com",
})
require.NoError(t, err)
creatorCtx := context.WithValue(ctx, auth.UserIDContextKey, creator.ID)
attachment, err := svc.CreateAttachment(creatorCtx, &apiv1.CreateAttachmentRequest{
Attachment: &apiv1.Attachment{
Filename: "motion.jpg",
Type: "image/jpeg",
Content: testutil.BuildMotionPhotoJPEG(),
},
})
require.NoError(t, err)
_, err = svc.CreateMemo(creatorCtx, &apiv1.CreateMemoRequest{
Memo: &apiv1.Memo{
Content: "motion memo",
Visibility: apiv1.Visibility_PUBLIC,
Attachments: []*apiv1.Attachment{
{Name: attachment.Name},
},
},
})
require.NoError(t, err)
e := echo.New()
fs.RegisterRoutes(e)
req := httptest.NewRequest(http.MethodGet, fmt.Sprintf("/file/%s/%s?motion=true", attachment.Name, attachment.Filename), nil)
rec := httptest.NewRecorder()
e.ServeHTTP(rec, req)
require.Equal(t, http.StatusOK, rec.Code)
require.Equal(t, "video/mp4", rec.Header().Get("Content-Type"))
require.Contains(t, rec.Body.String(), "ftyp")
}
func TestServeAttachmentFile_SVGThumbnailServedAsImageWithSecurityHeaders(t *testing.T) {
ctx := context.Background()
svc, fs, _, cleanup := newShareAttachmentTestServices(ctx, t)
defer cleanup()
creator, err := svc.Store.CreateUser(ctx, &store.User{
Username: "svg-owner",
Role: store.RoleUser,
Email: "svg-owner@example.com",
})
require.NoError(t, err)
creatorCtx := context.WithValue(ctx, auth.UserIDContextKey, creator.ID)
svgContent := []byte(`<svg xmlns="http://www.w3.org/2000/svg" width="120" height="40"><text x="0" y="20">memos</text></svg>`)
attachment, err := svc.CreateAttachment(creatorCtx, &apiv1.CreateAttachmentRequest{
Attachment: &apiv1.Attachment{
Filename: "preview.svg",
Type: "image/svg+xml",
Content: svgContent,
},
})
require.NoError(t, err)
_, err = svc.CreateMemo(creatorCtx, &apiv1.CreateMemoRequest{
Memo: &apiv1.Memo{
Content: "svg memo",
Visibility: apiv1.Visibility_PUBLIC,
Attachments: []*apiv1.Attachment{
{Name: attachment.Name},
},
},
})
require.NoError(t, err)
e := echo.New()
fs.RegisterRoutes(e)
req := httptest.NewRequest(http.MethodGet, fmt.Sprintf("/file/%s/%s?thumbnail=true", attachment.Name, attachment.Filename), nil)
rec := httptest.NewRecorder()
e.ServeHTTP(rec, req)
require.Equal(t, http.StatusOK, rec.Code)
require.Equal(t, "image/svg+xml", rec.Header().Get("Content-Type"))
require.Empty(t, rec.Header().Get("Content-Disposition"))
require.Equal(t, "nosniff", rec.Header().Get("X-Content-Type-Options"))
require.Equal(t, "default-src 'none'; style-src 'unsafe-inline';", rec.Header().Get("Content-Security-Policy"))
require.Equal(t, svgContent, rec.Body.Bytes())
}
func TestServeAttachmentFile_ThumbnailWithSensitiveMetadataServesOriginal(t *testing.T) {
ctx := context.Background()
svc, fs, _, cleanup := newShareAttachmentTestServices(ctx, t)
defer cleanup()
creator, err := svc.Store.CreateUser(ctx, &store.User{
Username: "hdr-owner",
Role: store.RoleUser,
Email: "hdr-owner@example.com",
})
require.NoError(t, err)
creatorCtx := context.WithValue(ctx, auth.UserIDContextKey, creator.ID)
imageContent := testPNGWithChunk(t, "cICP", []byte{9, 16, 9, 1})
attachment, err := svc.CreateAttachment(creatorCtx, &apiv1.CreateAttachmentRequest{
Attachment: &apiv1.Attachment{
Filename: "hdr.png",
Type: "image/png",
Content: imageContent,
},
})
require.NoError(t, err)
_, err = svc.CreateMemo(creatorCtx, &apiv1.CreateMemoRequest{
Memo: &apiv1.Memo{
Content: "hdr memo",
Visibility: apiv1.Visibility_PUBLIC,
Attachments: []*apiv1.Attachment{
{Name: attachment.Name},
},
},
})
require.NoError(t, err)
e := echo.New()
fs.RegisterRoutes(e)
req := httptest.NewRequest(http.MethodGet, fmt.Sprintf("/file/%s/%s?thumbnail=true", attachment.Name, attachment.Filename), nil)
rec := httptest.NewRecorder()
e.ServeHTTP(rec, req)
require.Equal(t, http.StatusOK, rec.Code)
require.Equal(t, "image/png", rec.Header().Get("Content-Type"))
require.Equal(t, imageContent, rec.Body.Bytes())
}
func TestHasThumbnailSensitiveMetadata(t *testing.T) {
tests := []struct {
name string
mimeType string
data []byte
want bool
}{
{
name: "jpeg hdr gain map",
mimeType: "image/jpeg",
data: []byte("xmp hdrgm:Version=\"1.0\""),
want: true,
},
{
name: "jpeg icc profile",
mimeType: "image/jpeg",
data: []byte("ICC_PROFILE"),
want: true,
},
{
name: "png cicp chunk",
mimeType: "image/png",
data: []byte("cICP"),
want: true,
},
{
name: "heic",
mimeType: "image/heic",
data: nil,
want: true,
},
{
name: "plain jpeg",
mimeType: "image/jpeg",
data: []byte("plain image data"),
want: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
require.Equal(t, tt.want, hasThumbnailSensitiveMetadata(tt.mimeType, tt.data))
})
}
}
func testPNGWithChunk(t *testing.T, chunkType string, chunkData []byte) []byte {
t.Helper()
img := image.NewRGBA(image.Rect(0, 0, 1, 1))
img.Set(0, 0, color.RGBA{R: 255, A: 255})
var encoded bytes.Buffer
require.NoError(t, png.Encode(&encoded, img))
pngData := encoded.Bytes()
iendIndex := bytes.LastIndex(pngData, []byte("IEND"))
require.GreaterOrEqual(t, iendIndex, 4)
chunkStart := iendIndex - 4
var chunk bytes.Buffer
require.NoError(t, binary.Write(&chunk, binary.BigEndian, uint32(len(chunkData))))
chunk.WriteString(chunkType)
chunk.Write(chunkData)
checksum := crc32.ChecksumIEEE(append([]byte(chunkType), chunkData...))
require.NoError(t, binary.Write(&chunk, binary.BigEndian, checksum))
result := make([]byte, 0, len(pngData)+chunk.Len())
result = append(result, pngData[:chunkStart]...)
result = append(result, chunk.Bytes()...)
result = append(result, pngData[chunkStart:]...)
return result
}
func newShareAttachmentTestServices(ctx context.Context, t *testing.T) (*apiv1service.APIV1Service, *FileServerService, *store.Store, func()) {
t.Helper()
testStore := teststore.NewTestingStore(ctx, t)
testProfile := &profile.Profile{
Demo: true,
Version: "test-1.0.0",
InstanceURL: "http://localhost:8080",
Driver: "sqlite",
DSN: ":memory:",
Data: t.TempDir(),
}
secret := "test-secret"
markdownService := markdown.NewService(markdown.WithTagExtension())
apiService := &apiv1service.APIV1Service{
Secret: secret,
Profile: testProfile,
Store: testStore,
MarkdownService: markdownService,
SSEHub: apiv1service.NewSSEHub(),
}
fileService := NewFileServerService(testProfile, testStore, secret)
return apiService, fileService, testStore, func() {
testStore.Close()
}
}