0.29.1原版
This commit is contained in:
@@ -0,0 +1,304 @@
|
||||
# Fileserver Package
|
||||
|
||||
## Overview
|
||||
|
||||
The `fileserver` package handles all binary file serving for Memos using native HTTP handlers. It was created to replace gRPC-based binary serving, which had limitations with HTTP range requests (required for Safari video/audio playback).
|
||||
|
||||
## Responsibilities
|
||||
|
||||
- Serve attachment binary files (images, videos, audio, documents)
|
||||
- Serve user avatar images
|
||||
- Handle HTTP range requests for video/audio streaming
|
||||
- Authenticate requests using JWT tokens or Personal Access Tokens
|
||||
- Check permissions for private content
|
||||
- Generate and serve image thumbnails
|
||||
- Prevent XSS attacks on uploaded content
|
||||
- Support S3 external storage
|
||||
|
||||
## Architecture
|
||||
|
||||
### Design Principles
|
||||
|
||||
1. **Separation of Concerns**: Binary files via HTTP, metadata via gRPC
|
||||
2. **DRY**: Imports auth constants from `api/v1` package (single source of truth)
|
||||
3. **Security First**: Authentication, authorization, and XSS prevention
|
||||
4. **Performance**: Native HTTP streaming with proper caching headers
|
||||
|
||||
### Package Structure
|
||||
|
||||
```
|
||||
fileserver/
|
||||
├── fileserver.go # Main service and HTTP handlers
|
||||
├── README.md # This file
|
||||
└── fileserver_test.go # Tests (to be added)
|
||||
```
|
||||
|
||||
## API Endpoints
|
||||
|
||||
### 1. Attachment Binary
|
||||
```
|
||||
GET /file/attachments/:uid/:filename[?thumbnail=true]
|
||||
```
|
||||
|
||||
**Parameters:**
|
||||
- `uid` - Attachment unique identifier
|
||||
- `filename` - Original filename
|
||||
- `thumbnail` (optional) - Return thumbnail for images
|
||||
|
||||
**Authentication:** Required for non-public memos
|
||||
|
||||
**Response:**
|
||||
- `200 OK` - File content with proper Content-Type
|
||||
- `206 Partial Content` - For range requests (video/audio)
|
||||
- `401 Unauthorized` - Authentication required
|
||||
- `403 Forbidden` - User not authorized
|
||||
- `404 Not Found` - Attachment not found
|
||||
|
||||
**Headers:**
|
||||
- `Content-Type` - MIME type of the file
|
||||
- `Cache-Control: public, max-age=3600`
|
||||
- `Accept-Ranges: bytes` - For video/audio
|
||||
- `Content-Range` - For partial responses (206)
|
||||
|
||||
### 2. User Avatar
|
||||
```
|
||||
GET /file/users/:identifier/avatar
|
||||
```
|
||||
|
||||
**Parameters:**
|
||||
- `identifier` - User ID (e.g., `1`) or username (e.g., `steven`)
|
||||
|
||||
**Authentication:** Not required (avatars are public)
|
||||
|
||||
**Response:**
|
||||
- `200 OK` - Avatar image (PNG/JPEG)
|
||||
- `404 Not Found` - User not found or no avatar set
|
||||
|
||||
**Headers:**
|
||||
- `Content-Type` - image/png or image/jpeg
|
||||
- `Cache-Control: public, max-age=3600`
|
||||
|
||||
## Authentication
|
||||
|
||||
### Supported Methods
|
||||
|
||||
The fileserver supports the following authentication methods:
|
||||
|
||||
1. **JWT Access Token** (`Authorization: Bearer {token}`)
|
||||
- Short-lived tokens (15 minutes) for API access
|
||||
- Stateless validation using JWT signature
|
||||
- Extracts user ID from token claims
|
||||
|
||||
2. **Personal Access Token (PAT)** (`Authorization: Bearer {pat}`)
|
||||
- Long-lived tokens for programmatic access
|
||||
- Validates against database for revocation
|
||||
- Prefixed with specific identifier
|
||||
|
||||
### Authentication Flow
|
||||
|
||||
```
|
||||
Request → getCurrentUser()
|
||||
├─→ Try Session Cookie
|
||||
│ ├─→ Parse cookie value
|
||||
│ ├─→ Get user from DB
|
||||
│ ├─→ Validate session
|
||||
│ └─→ Return user (if valid)
|
||||
│
|
||||
└─→ Try JWT Token
|
||||
├─→ Parse Authorization header
|
||||
├─→ Verify JWT signature
|
||||
├─→ Get user from DB
|
||||
├─→ Validate token in access tokens list
|
||||
└─→ Return user (if valid)
|
||||
```
|
||||
|
||||
### Permission Model
|
||||
|
||||
**Attachments:**
|
||||
- Unlinked: Public (no auth required)
|
||||
- Public memo: Public (no auth required)
|
||||
- Protected memo: Requires authentication
|
||||
- Private memo: Creator only
|
||||
|
||||
**Avatars:**
|
||||
- Always public (no auth required)
|
||||
|
||||
## Key Functions
|
||||
|
||||
### HTTP Handlers
|
||||
|
||||
#### `serveAttachmentFile(c echo.Context) error`
|
||||
Main handler for attachment binary serving.
|
||||
|
||||
**Flow:**
|
||||
1. Extract UID from URL parameter
|
||||
2. Fetch attachment from database
|
||||
3. Check permissions (memo visibility)
|
||||
4. Get binary blob (local file, S3, or database)
|
||||
5. Handle thumbnail request (if applicable)
|
||||
6. Set security headers (XSS prevention)
|
||||
7. Serve with range request support (video/audio)
|
||||
|
||||
#### `serveUserAvatar(c echo.Context) error`
|
||||
Main handler for user avatar serving.
|
||||
|
||||
**Flow:**
|
||||
1. Extract identifier (ID or username) from URL
|
||||
2. Lookup user in database
|
||||
3. Check if avatar exists
|
||||
4. Decode base64 data URI
|
||||
5. Serve with proper content type and caching
|
||||
|
||||
### Authentication
|
||||
|
||||
#### `getCurrentUser(ctx, c) (*store.User, error)`
|
||||
Authenticates request using session cookie or JWT token.
|
||||
|
||||
#### `authenticateBySession(ctx, cookie) (*store.User, error)`
|
||||
Validates session cookie and returns authenticated user.
|
||||
|
||||
#### `authenticateByJWT(ctx, token) (*store.User, error)`
|
||||
Validates JWT access token and returns authenticated user.
|
||||
|
||||
### Permission Checks
|
||||
|
||||
#### `checkAttachmentPermission(ctx, c, attachment) error`
|
||||
Validates user has permission to access attachment based on memo visibility.
|
||||
|
||||
### File Operations
|
||||
|
||||
#### `getAttachmentBlob(attachment) ([]byte, error)`
|
||||
Retrieves binary content from local storage, S3, or database.
|
||||
|
||||
#### `getOrGenerateThumbnail(ctx, attachment) ([]byte, error)`
|
||||
Returns cached thumbnail or generates new one (with semaphore limiting).
|
||||
|
||||
### Utilities
|
||||
|
||||
#### `getUserByIdentifier(ctx, identifier) (*store.User, error)`
|
||||
Finds user by ID (int) or username (string).
|
||||
|
||||
#### `extractImageInfo(dataURI) (type, base64, error)`
|
||||
Parses data URI to extract MIME type and base64 data.
|
||||
|
||||
## Dependencies
|
||||
|
||||
### External Packages
|
||||
- `github.com/labstack/echo/v5` - HTTP router and middleware
|
||||
- `github.com/golang-jwt/jwt/v5` - JWT parsing and validation
|
||||
- `github.com/disintegration/imaging` - Image thumbnail generation
|
||||
- `golang.org/x/sync/semaphore` - Concurrency control for thumbnails
|
||||
|
||||
### Internal Packages
|
||||
- `server/auth` - Authentication utilities
|
||||
- `store` - Database operations
|
||||
- `internal/profile` - Server configuration
|
||||
- `internal/storage/s3` - S3 storage client
|
||||
|
||||
## Configuration
|
||||
|
||||
### Constants
|
||||
|
||||
Auth-related constants are imported from `server/auth`:
|
||||
- `auth.RefreshTokenCookieName` - "memos_refresh"
|
||||
- `auth.PersonalAccessTokenPrefix` - PAT identifier prefix
|
||||
|
||||
Package-specific constants:
|
||||
- `ThumbnailCacheFolder` - ".thumbnail_cache"
|
||||
- `thumbnailMaxSize` - 600px
|
||||
- `SupportedThumbnailMimeTypes` - ["image/png", "image/jpeg"]
|
||||
|
||||
## Error Handling
|
||||
|
||||
All handlers return Echo HTTP errors with appropriate status codes:
|
||||
|
||||
```go
|
||||
// Bad request
|
||||
echo.NewHTTPError(http.StatusBadRequest, "message")
|
||||
|
||||
// Unauthorized (no auth)
|
||||
echo.NewHTTPError(http.StatusUnauthorized, "message")
|
||||
|
||||
// Forbidden (auth but no permission)
|
||||
echo.NewHTTPError(http.StatusForbidden, "message")
|
||||
|
||||
// Not found
|
||||
echo.NewHTTPError(http.StatusNotFound, "message")
|
||||
|
||||
// Internal error
|
||||
echo.NewHTTPError(http.StatusInternalServerError, "message").SetInternal(err)
|
||||
```
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### 1. XSS Prevention
|
||||
SVG and HTML files are served as `application/octet-stream` to prevent script execution:
|
||||
|
||||
```go
|
||||
if contentType == "image/svg+xml" ||
|
||||
contentType == "text/html" ||
|
||||
contentType == "application/xhtml+xml" {
|
||||
contentType = "application/octet-stream"
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Authentication
|
||||
Private content requires valid JWT access token or Personal Access Token.
|
||||
|
||||
### 3. Authorization
|
||||
Memo visibility rules enforced before serving attachments.
|
||||
|
||||
### 4. Input Validation
|
||||
- Attachment UID validated from database
|
||||
- User identifier validated (ID or username)
|
||||
- Range requests validated before processing
|
||||
|
||||
## Performance Optimizations
|
||||
|
||||
### 1. Thumbnail Caching
|
||||
Thumbnails cached on disk to avoid regeneration:
|
||||
- Cache location: `{data_dir}/.thumbnail_cache/`
|
||||
- Filename: `{attachment_id}{extension}`
|
||||
- Semaphore limits concurrent generation (max 3)
|
||||
|
||||
### 2. HTTP Range Requests
|
||||
Video/audio files use `http.ServeContent()` for efficient streaming:
|
||||
- Automatic range parsing
|
||||
- Efficient memory usage (streaming, not loading full file)
|
||||
- Safari-compatible partial content responses
|
||||
|
||||
### 3. Caching Headers
|
||||
All responses include cache headers:
|
||||
```
|
||||
Cache-Control: public, max-age=3600
|
||||
```
|
||||
|
||||
### 4. S3 External Links
|
||||
S3 files served via presigned URLs (no server download).
|
||||
|
||||
## Testing
|
||||
|
||||
### Unit Tests (To Add)
|
||||
See SAFARI_FIX.md for recommended test coverage.
|
||||
|
||||
### Manual Testing
|
||||
```bash
|
||||
# Test attachment
|
||||
curl "http://localhost:8081/file/attachments/{uid}/file.jpg"
|
||||
|
||||
# Test avatar by username
|
||||
curl "http://localhost:8081/file/users/steven/avatar"
|
||||
|
||||
# Test range request
|
||||
curl -H "Range: bytes=0-999" "http://localhost:8081/file/attachments/{uid}/video.mp4"
|
||||
```
|
||||
|
||||
## Future Improvements
|
||||
|
||||
See SAFARI_FIX.md section "Future Improvements" for planned enhancements.
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [SAFARI_FIX.md](../../../SAFARI_FIX.md) - Full migration guide
|
||||
- [server/router/api/v1/auth.go](../api/v1/auth.go) - Auth constants source of truth
|
||||
- [RFC 7233](https://tools.ietf.org/html/rfc7233) - HTTP Range Requests spec
|
||||
@@ -0,0 +1,764 @@
|
||||
package fileserver
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/disintegration/imaging"
|
||||
"github.com/labstack/echo/v5"
|
||||
"github.com/pkg/errors"
|
||||
"golang.org/x/sync/semaphore"
|
||||
|
||||
"github.com/usememos/memos/internal/motionphoto"
|
||||
"github.com/usememos/memos/internal/profile"
|
||||
"github.com/usememos/memos/internal/storage/s3"
|
||||
storepb "github.com/usememos/memos/proto/gen/store"
|
||||
"github.com/usememos/memos/server/auth"
|
||||
"github.com/usememos/memos/store"
|
||||
)
|
||||
|
||||
// Constants for file serving configuration.
|
||||
const (
|
||||
// ThumbnailCacheFolder is the folder name where thumbnail images are stored.
|
||||
ThumbnailCacheFolder = ".thumbnail_cache"
|
||||
|
||||
// MotionCacheFolder is the folder name where extracted motion clips are stored.
|
||||
MotionCacheFolder = ".motion_cache"
|
||||
|
||||
// thumbnailMaxSize is the maximum dimension (width or height) for thumbnails.
|
||||
thumbnailMaxSize = 600
|
||||
|
||||
// thumbnailMetadataProbeSize is the maximum number of original image bytes inspected
|
||||
// before thumbnail generation to detect metadata that the JPEG thumbnail pipeline cannot preserve.
|
||||
thumbnailMetadataProbeSize = 1 << 20
|
||||
|
||||
// maxConcurrentThumbnails limits concurrent thumbnail generation to prevent memory exhaustion.
|
||||
maxConcurrentThumbnails = 3
|
||||
|
||||
// cacheMaxAge is the max-age value for Cache-Control headers (1 hour).
|
||||
cacheMaxAge = "public, max-age=3600"
|
||||
)
|
||||
|
||||
// xssUnsafeTypes contains MIME types that could execute scripts if served directly.
|
||||
// These are served as application/octet-stream to prevent XSS attacks.
|
||||
var xssUnsafeTypes = map[string]bool{
|
||||
"text/html": true,
|
||||
"text/javascript": true,
|
||||
"application/javascript": true,
|
||||
"application/x-javascript": true,
|
||||
"text/xml": true,
|
||||
"application/xml": true,
|
||||
"application/xhtml+xml": true,
|
||||
}
|
||||
|
||||
// thumbnailSupportedTypes contains image MIME types that support thumbnail generation.
|
||||
var thumbnailSupportedTypes = map[string]bool{
|
||||
"image/png": true,
|
||||
"image/jpeg": true,
|
||||
"image/jpg": true,
|
||||
"image/heic": true,
|
||||
"image/heif": true,
|
||||
"image/webp": true,
|
||||
}
|
||||
|
||||
// avatarAllowedTypes contains MIME types allowed for user avatars.
|
||||
var avatarAllowedTypes = map[string]bool{
|
||||
"image/png": true,
|
||||
"image/jpeg": true,
|
||||
"image/jpg": true,
|
||||
"image/gif": true,
|
||||
"image/webp": true,
|
||||
"image/heic": true,
|
||||
"image/heif": true,
|
||||
}
|
||||
|
||||
// SupportedThumbnailMimeTypes is the exported list of thumbnail-supported MIME types.
|
||||
var SupportedThumbnailMimeTypes = []string{
|
||||
"image/png",
|
||||
"image/jpeg",
|
||||
"image/jpg",
|
||||
"image/heic",
|
||||
"image/heif",
|
||||
"image/webp",
|
||||
}
|
||||
|
||||
var errUseOriginalForThumbnail = errors.New("serve original image instead of metadata-stripping thumbnail")
|
||||
|
||||
// dataURIRegex parses data URI format: data:image/png;base64,iVBORw0KGgo...
|
||||
var dataURIRegex = regexp.MustCompile(`^data:(?P<type>[^;]+);base64,(?P<base64>.+)`)
|
||||
|
||||
// FileServerService handles HTTP file serving with proper range request support.
|
||||
type FileServerService struct {
|
||||
Profile *profile.Profile
|
||||
Store *store.Store
|
||||
authenticator *auth.Authenticator
|
||||
|
||||
// thumbnailSemaphore limits concurrent thumbnail generation.
|
||||
thumbnailSemaphore *semaphore.Weighted
|
||||
}
|
||||
|
||||
// NewFileServerService creates a new file server service.
|
||||
func NewFileServerService(profile *profile.Profile, store *store.Store, secret string) *FileServerService {
|
||||
return &FileServerService{
|
||||
Profile: profile,
|
||||
Store: store,
|
||||
authenticator: auth.NewAuthenticator(store, secret),
|
||||
thumbnailSemaphore: semaphore.NewWeighted(maxConcurrentThumbnails),
|
||||
}
|
||||
}
|
||||
|
||||
// RegisterRoutes registers HTTP file serving routes.
|
||||
func (s *FileServerService) RegisterRoutes(echoServer *echo.Echo) {
|
||||
fileGroup := echoServer.Group("/file")
|
||||
fileGroup.GET("/attachments/:uid/:filename", s.serveAttachmentFile)
|
||||
fileGroup.GET("/users/:identifier/avatar", s.serveUserAvatar)
|
||||
}
|
||||
|
||||
// =============================================================================
|
||||
// HTTP Handlers
|
||||
// =============================================================================
|
||||
|
||||
// serveAttachmentFile serves attachment binary content using native HTTP.
|
||||
func (s *FileServerService) serveAttachmentFile(c *echo.Context) error {
|
||||
ctx := c.Request().Context()
|
||||
uid := c.Param("uid")
|
||||
wantThumbnail := c.QueryParam("thumbnail") == "true"
|
||||
wantMotion := c.QueryParam("motion") == "true"
|
||||
|
||||
attachment, err := s.Store.GetAttachment(ctx, &store.FindAttachment{
|
||||
UID: &uid,
|
||||
GetBlob: true,
|
||||
})
|
||||
if err != nil {
|
||||
return echo.NewHTTPError(http.StatusInternalServerError, "failed to get attachment").Wrap(err)
|
||||
}
|
||||
if attachment == nil {
|
||||
return echo.NewHTTPError(http.StatusNotFound, "attachment not found")
|
||||
}
|
||||
|
||||
if err := s.checkAttachmentPermission(ctx, c, attachment); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if wantMotion {
|
||||
return s.serveMotionClip(c, attachment)
|
||||
}
|
||||
|
||||
contentType := s.sanitizeContentType(attachment.Type)
|
||||
|
||||
// Stream video/audio to avoid loading entire file into memory.
|
||||
if isMediaType(attachment.Type) {
|
||||
return s.serveMediaStream(c, attachment, contentType)
|
||||
}
|
||||
|
||||
return s.serveStaticFile(c, attachment, contentType, wantThumbnail)
|
||||
}
|
||||
|
||||
// serveUserAvatar serves user avatar images.
|
||||
func (s *FileServerService) serveUserAvatar(c *echo.Context) error {
|
||||
ctx := c.Request().Context()
|
||||
identifier := c.Param("identifier")
|
||||
|
||||
user, err := s.getUserByUsername(ctx, identifier)
|
||||
if err != nil {
|
||||
return echo.NewHTTPError(http.StatusInternalServerError, "failed to get user").Wrap(err)
|
||||
}
|
||||
if user == nil {
|
||||
return echo.NewHTTPError(http.StatusNotFound, "user not found")
|
||||
}
|
||||
if user.AvatarURL == "" {
|
||||
return echo.NewHTTPError(http.StatusNotFound, "avatar not found")
|
||||
}
|
||||
|
||||
imageType, imageData, err := s.parseDataURI(user.AvatarURL)
|
||||
if err != nil {
|
||||
return echo.NewHTTPError(http.StatusInternalServerError, "failed to parse avatar data").Wrap(err)
|
||||
}
|
||||
|
||||
if !avatarAllowedTypes[imageType] {
|
||||
return echo.NewHTTPError(http.StatusBadRequest, "invalid avatar image type")
|
||||
}
|
||||
|
||||
setSecurityHeaders(c)
|
||||
c.Response().Header().Set(echo.HeaderContentType, imageType)
|
||||
c.Response().Header().Set(echo.HeaderCacheControl, cacheMaxAge)
|
||||
|
||||
return c.Blob(http.StatusOK, imageType, imageData)
|
||||
}
|
||||
|
||||
// =============================================================================
|
||||
// File Serving Methods
|
||||
// =============================================================================
|
||||
|
||||
// serveMediaStream serves video/audio files using streaming to avoid memory exhaustion.
|
||||
func (s *FileServerService) serveMediaStream(c *echo.Context, attachment *store.Attachment, contentType string) error {
|
||||
setSecurityHeaders(c)
|
||||
setMediaHeaders(c, contentType, attachment.Type)
|
||||
|
||||
switch attachment.StorageType {
|
||||
case storepb.AttachmentStorageType_LOCAL:
|
||||
filePath, err := s.resolveLocalPath(attachment.Reference)
|
||||
if err != nil {
|
||||
return echo.NewHTTPError(http.StatusInternalServerError, "failed to resolve file path").Wrap(err)
|
||||
}
|
||||
http.ServeFile(c.Response(), c.Request(), filePath)
|
||||
return nil
|
||||
|
||||
case storepb.AttachmentStorageType_S3:
|
||||
presignURL, err := s.getS3PresignedURL(c.Request().Context(), attachment)
|
||||
if err != nil {
|
||||
return echo.NewHTTPError(http.StatusInternalServerError, "failed to generate presigned URL").Wrap(err)
|
||||
}
|
||||
return c.Redirect(http.StatusTemporaryRedirect, presignURL)
|
||||
|
||||
default:
|
||||
// Database storage fallback.
|
||||
modTime := time.Unix(attachment.UpdatedTs, 0)
|
||||
http.ServeContent(c.Response(), c.Request(), attachment.Filename, modTime, bytes.NewReader(attachment.Blob))
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
// serveStaticFile serves non-streaming files (images, documents, etc.).
|
||||
func (s *FileServerService) serveStaticFile(c *echo.Context, attachment *store.Attachment, contentType string, wantThumbnail bool) error {
|
||||
// Generate thumbnail for supported image types.
|
||||
if wantThumbnail && thumbnailSupportedTypes[attachment.Type] {
|
||||
if thumbnailBlob, err := s.getOrGenerateThumbnail(c.Request().Context(), attachment); err != nil {
|
||||
if !errors.Is(err, errUseOriginalForThumbnail) {
|
||||
slog.Warn("failed to get thumbnail", "error", err)
|
||||
}
|
||||
} else {
|
||||
setSecurityHeaders(c)
|
||||
setMediaHeaders(c, "image/jpeg", attachment.Type)
|
||||
return c.Blob(http.StatusOK, "image/jpeg", thumbnailBlob)
|
||||
}
|
||||
}
|
||||
|
||||
setSecurityHeaders(c)
|
||||
setMediaHeaders(c, contentType, attachment.Type)
|
||||
|
||||
// Force download for non-media files to prevent XSS execution.
|
||||
if !strings.HasPrefix(contentType, "image/") && contentType != "application/pdf" {
|
||||
c.Response().Header().Set(echo.HeaderContentDisposition, fmt.Sprintf("attachment; filename=%q", attachment.Filename))
|
||||
}
|
||||
|
||||
switch attachment.StorageType {
|
||||
case storepb.AttachmentStorageType_LOCAL:
|
||||
filePath, err := s.resolveLocalPath(attachment.Reference)
|
||||
if err != nil {
|
||||
return echo.NewHTTPError(http.StatusInternalServerError, "failed to resolve file path").Wrap(err)
|
||||
}
|
||||
http.ServeFile(c.Response(), c.Request(), filePath)
|
||||
return nil
|
||||
case storepb.AttachmentStorageType_S3:
|
||||
reader, err := s.getAttachmentReader(c.Request().Context(), attachment)
|
||||
if err != nil {
|
||||
return echo.NewHTTPError(http.StatusInternalServerError, "failed to get attachment reader").Wrap(err)
|
||||
}
|
||||
defer reader.Close()
|
||||
return c.Stream(http.StatusOK, contentType, reader)
|
||||
default:
|
||||
return c.Blob(http.StatusOK, contentType, attachment.Blob)
|
||||
}
|
||||
}
|
||||
|
||||
// =============================================================================
|
||||
// Storage Operations
|
||||
// =============================================================================
|
||||
|
||||
// getAttachmentBlob retrieves the binary content of an attachment from storage.
|
||||
func (s *FileServerService) getAttachmentBlob(attachment *store.Attachment) ([]byte, error) {
|
||||
switch attachment.StorageType {
|
||||
case storepb.AttachmentStorageType_LOCAL:
|
||||
return s.readLocalFile(attachment.Reference)
|
||||
|
||||
case storepb.AttachmentStorageType_S3:
|
||||
return s.downloadFromS3(context.Background(), attachment)
|
||||
|
||||
default:
|
||||
return attachment.Blob, nil
|
||||
}
|
||||
}
|
||||
|
||||
// getAttachmentReader returns a reader for streaming attachment content.
|
||||
func (s *FileServerService) getAttachmentReader(ctx context.Context, attachment *store.Attachment) (io.ReadCloser, error) {
|
||||
switch attachment.StorageType {
|
||||
case storepb.AttachmentStorageType_LOCAL:
|
||||
filePath, err := s.resolveLocalPath(attachment.Reference)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
file, err := os.Open(filePath)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return nil, errors.Wrap(err, "file not found")
|
||||
}
|
||||
return nil, errors.Wrap(err, "failed to open file")
|
||||
}
|
||||
return file, nil
|
||||
|
||||
case storepb.AttachmentStorageType_S3:
|
||||
s3Client, s3Object, err := s.createS3Client(attachment)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
reader, err := s3Client.GetObjectStream(ctx, s3Object.Key)
|
||||
if err != nil {
|
||||
return nil, errors.Wrap(err, "failed to stream from S3")
|
||||
}
|
||||
return reader, nil
|
||||
|
||||
default:
|
||||
return io.NopCloser(bytes.NewReader(attachment.Blob)), nil
|
||||
}
|
||||
}
|
||||
|
||||
// resolveLocalPath converts a storage reference to an absolute file path.
|
||||
func (s *FileServerService) resolveLocalPath(reference string) (string, error) {
|
||||
filePath := filepath.FromSlash(reference)
|
||||
if !filepath.IsAbs(filePath) {
|
||||
filePath = filepath.Join(s.Profile.Data, filePath)
|
||||
}
|
||||
return filePath, nil
|
||||
}
|
||||
|
||||
// readLocalFile reads the entire contents of a local file.
|
||||
func (s *FileServerService) readLocalFile(reference string) ([]byte, error) {
|
||||
filePath, err := s.resolveLocalPath(reference)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
file, err := os.Open(filePath)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return nil, errors.Wrap(err, "file not found")
|
||||
}
|
||||
return nil, errors.Wrap(err, "failed to open file")
|
||||
}
|
||||
defer file.Close()
|
||||
|
||||
blob, err := io.ReadAll(file)
|
||||
if err != nil {
|
||||
return nil, errors.Wrap(err, "failed to read file")
|
||||
}
|
||||
return blob, nil
|
||||
}
|
||||
|
||||
// createS3Client creates an S3 client from attachment payload.
|
||||
func (*FileServerService) createS3Client(attachment *store.Attachment) (*s3.Client, *storepb.AttachmentPayload_S3Object, error) {
|
||||
if attachment.Payload == nil {
|
||||
return nil, nil, errors.New("attachment payload is missing")
|
||||
}
|
||||
s3Object := attachment.Payload.GetS3Object()
|
||||
if s3Object == nil {
|
||||
return nil, nil, errors.New("S3 object payload is missing")
|
||||
}
|
||||
if s3Object.S3Config == nil {
|
||||
return nil, nil, errors.New("S3 config is missing")
|
||||
}
|
||||
if s3Object.Key == "" {
|
||||
return nil, nil, errors.New("S3 object key is missing")
|
||||
}
|
||||
|
||||
client, err := s3.NewClient(context.Background(), s3Object.S3Config)
|
||||
if err != nil {
|
||||
return nil, nil, errors.Wrap(err, "failed to create S3 client")
|
||||
}
|
||||
return client, s3Object, nil
|
||||
}
|
||||
|
||||
// downloadFromS3 downloads the entire object from S3.
|
||||
func (s *FileServerService) downloadFromS3(ctx context.Context, attachment *store.Attachment) ([]byte, error) {
|
||||
client, s3Object, err := s.createS3Client(attachment)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
blob, err := client.GetObject(ctx, s3Object.Key)
|
||||
if err != nil {
|
||||
return nil, errors.Wrap(err, "failed to download from S3")
|
||||
}
|
||||
return blob, nil
|
||||
}
|
||||
|
||||
// getS3PresignedURL generates a presigned URL for direct S3 access.
|
||||
func (s *FileServerService) getS3PresignedURL(ctx context.Context, attachment *store.Attachment) (string, error) {
|
||||
client, s3Object, err := s.createS3Client(attachment)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
url, err := client.PresignGetObject(ctx, s3Object.Key)
|
||||
if err != nil {
|
||||
return "", errors.Wrap(err, "failed to presign URL")
|
||||
}
|
||||
return url, nil
|
||||
}
|
||||
|
||||
// =============================================================================
|
||||
// Thumbnail Generation
|
||||
// =============================================================================
|
||||
|
||||
// getOrGenerateThumbnail returns the thumbnail image of the attachment.
|
||||
// Uses semaphore to limit concurrent thumbnail generation and prevent memory exhaustion.
|
||||
func (s *FileServerService) getOrGenerateThumbnail(ctx context.Context, attachment *store.Attachment) ([]byte, error) {
|
||||
thumbnailPath, err := s.getThumbnailPath(attachment)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Fast path: return cached thumbnail if exists.
|
||||
if blob, err := s.readCachedThumbnail(thumbnailPath); err == nil {
|
||||
return blob, nil
|
||||
}
|
||||
|
||||
useOriginal, err := s.shouldUseOriginalForThumbnail(ctx, attachment)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if useOriginal {
|
||||
return nil, errUseOriginalForThumbnail
|
||||
}
|
||||
|
||||
// Acquire semaphore to limit concurrent generation.
|
||||
if err := s.thumbnailSemaphore.Acquire(ctx, 1); err != nil {
|
||||
return nil, errors.Wrap(err, "failed to acquire semaphore")
|
||||
}
|
||||
defer s.thumbnailSemaphore.Release(1)
|
||||
|
||||
// Double-check after acquiring semaphore (another goroutine may have generated it).
|
||||
if blob, err := s.readCachedThumbnail(thumbnailPath); err == nil {
|
||||
return blob, nil
|
||||
}
|
||||
|
||||
return s.generateThumbnail(ctx, attachment, thumbnailPath)
|
||||
}
|
||||
|
||||
// getThumbnailPath returns the file path for a cached thumbnail.
|
||||
func (s *FileServerService) getThumbnailPath(attachment *store.Attachment) (string, error) {
|
||||
cacheFolder := filepath.Join(s.Profile.Data, ThumbnailCacheFolder)
|
||||
if err := os.MkdirAll(cacheFolder, os.ModePerm); err != nil {
|
||||
return "", errors.Wrap(err, "failed to create thumbnail cache folder")
|
||||
}
|
||||
filename := fmt.Sprintf("%s.v2.jpeg", attachment.UID)
|
||||
return filepath.Join(cacheFolder, filename), nil
|
||||
}
|
||||
|
||||
func (s *FileServerService) shouldUseOriginalForThumbnail(ctx context.Context, attachment *store.Attachment) (bool, error) {
|
||||
if attachment.Type == "image/heic" || attachment.Type == "image/heif" {
|
||||
return true, nil
|
||||
}
|
||||
|
||||
if attachment.Type != "image/jpeg" && attachment.Type != "image/jpg" && attachment.Type != "image/png" && attachment.Type != "image/webp" {
|
||||
return false, nil
|
||||
}
|
||||
|
||||
reader, err := s.getAttachmentReader(ctx, attachment)
|
||||
if err != nil {
|
||||
return false, errors.Wrap(err, "failed to open image for metadata probe")
|
||||
}
|
||||
defer reader.Close()
|
||||
|
||||
probe, err := io.ReadAll(io.LimitReader(reader, thumbnailMetadataProbeSize))
|
||||
if err != nil {
|
||||
return false, errors.Wrap(err, "failed to read image metadata probe")
|
||||
}
|
||||
|
||||
return hasThumbnailSensitiveMetadata(attachment.Type, probe), nil
|
||||
}
|
||||
|
||||
func hasThumbnailSensitiveMetadata(mimeType string, data []byte) bool {
|
||||
if mimeType == "image/heic" || mimeType == "image/heif" {
|
||||
return true
|
||||
}
|
||||
|
||||
for _, marker := range [][]byte{
|
||||
[]byte("ICC_PROFILE"),
|
||||
[]byte("iCCP"),
|
||||
[]byte("ICCP"),
|
||||
[]byte("cICP"),
|
||||
[]byte("mDCv"),
|
||||
[]byte("cLLi"),
|
||||
} {
|
||||
if bytes.Contains(data, marker) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
|
||||
lowerData := strings.ToLower(string(data))
|
||||
for _, marker := range []string{
|
||||
"hdrgm:",
|
||||
"hdr gain map",
|
||||
"hdrgainmap",
|
||||
"gainmap",
|
||||
"ultrahdr",
|
||||
"adobe:hdrgainmap",
|
||||
"aux:hdr",
|
||||
"auxiliaryimagetype",
|
||||
"display p3",
|
||||
"display-p3",
|
||||
"rec.2020",
|
||||
"bt.2020",
|
||||
"arib-std-b67",
|
||||
"smpte st 2084",
|
||||
} {
|
||||
if strings.Contains(lowerData, marker) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
// readCachedThumbnail reads a thumbnail from the cache directory.
|
||||
func (*FileServerService) readCachedThumbnail(path string) ([]byte, error) {
|
||||
file, err := os.Open(path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer file.Close()
|
||||
return io.ReadAll(file)
|
||||
}
|
||||
|
||||
// generateThumbnail creates a new thumbnail and saves it to disk.
|
||||
func (s *FileServerService) generateThumbnail(ctx context.Context, attachment *store.Attachment, thumbnailPath string) ([]byte, error) {
|
||||
reader, err := s.getAttachmentReader(ctx, attachment)
|
||||
if err != nil {
|
||||
return nil, errors.Wrap(err, "failed to get attachment reader")
|
||||
}
|
||||
defer reader.Close()
|
||||
|
||||
img, err := imaging.Decode(reader, imaging.AutoOrientation(true))
|
||||
if err != nil {
|
||||
return nil, errors.Wrap(err, "failed to decode image")
|
||||
}
|
||||
|
||||
width, height := img.Bounds().Dx(), img.Bounds().Dy()
|
||||
thumbnailWidth, thumbnailHeight := calculateThumbnailDimensions(width, height)
|
||||
|
||||
thumbnailImage := imaging.Resize(img, thumbnailWidth, thumbnailHeight, imaging.Lanczos)
|
||||
|
||||
output, err := os.Create(thumbnailPath)
|
||||
if err != nil {
|
||||
return nil, errors.Wrap(err, "failed to create thumbnail file")
|
||||
}
|
||||
defer output.Close()
|
||||
|
||||
if err := imaging.Encode(output, thumbnailImage, imaging.JPEG, imaging.JPEGQuality(90)); err != nil {
|
||||
return nil, errors.Wrap(err, "failed to save thumbnail")
|
||||
}
|
||||
|
||||
return s.readCachedThumbnail(thumbnailPath)
|
||||
}
|
||||
|
||||
// calculateThumbnailDimensions calculates the target dimensions for a thumbnail.
|
||||
// The largest dimension is constrained to thumbnailMaxSize while maintaining aspect ratio.
|
||||
// Small images are not enlarged.
|
||||
func calculateThumbnailDimensions(width, height int) (int, int) {
|
||||
if max(width, height) <= thumbnailMaxSize {
|
||||
return width, height
|
||||
}
|
||||
if width >= height {
|
||||
return thumbnailMaxSize, 0 // Landscape: constrain width.
|
||||
}
|
||||
return 0, thumbnailMaxSize // Portrait: constrain height.
|
||||
}
|
||||
|
||||
func (s *FileServerService) serveMotionClip(c *echo.Context, attachment *store.Attachment) error {
|
||||
motionMedia := attachment.Payload.GetMotionMedia()
|
||||
if motionMedia == nil || motionMedia.Family != storepb.MotionMediaFamily_ANDROID_MOTION_PHOTO || !motionMedia.HasEmbeddedVideo {
|
||||
return echo.NewHTTPError(http.StatusBadRequest, "attachment does not have motion clip")
|
||||
}
|
||||
|
||||
clipBlob, err := s.getOrExtractMotionClip(c.Request().Context(), attachment)
|
||||
if err != nil {
|
||||
return echo.NewHTTPError(http.StatusInternalServerError, "failed to get motion clip").Wrap(err)
|
||||
}
|
||||
|
||||
setSecurityHeaders(c)
|
||||
setMediaHeaders(c, "video/mp4", "video/mp4")
|
||||
modTime := time.Unix(attachment.UpdatedTs, 0)
|
||||
http.ServeContent(c.Response(), c.Request(), attachment.UID+".mp4", modTime, bytes.NewReader(clipBlob))
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *FileServerService) getOrExtractMotionClip(_ context.Context, attachment *store.Attachment) ([]byte, error) {
|
||||
motionPath, err := s.getMotionPath(attachment)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if blob, err := s.readCachedThumbnail(motionPath); err == nil {
|
||||
return blob, nil
|
||||
}
|
||||
|
||||
blob, err := s.getAttachmentBlob(attachment)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
videoBlob, _ := motionphoto.ExtractVideo(blob)
|
||||
if len(videoBlob) == 0 {
|
||||
return nil, errors.New("motion video not found")
|
||||
}
|
||||
|
||||
if err := os.WriteFile(motionPath, videoBlob, 0644); err != nil {
|
||||
return nil, errors.Wrap(err, "failed to cache motion clip")
|
||||
}
|
||||
|
||||
return videoBlob, nil
|
||||
}
|
||||
|
||||
func (s *FileServerService) getMotionPath(attachment *store.Attachment) (string, error) {
|
||||
cacheFolder := filepath.Join(s.Profile.Data, MotionCacheFolder)
|
||||
if err := os.MkdirAll(cacheFolder, os.ModePerm); err != nil {
|
||||
return "", errors.Wrap(err, "failed to create motion cache folder")
|
||||
}
|
||||
|
||||
return filepath.Join(cacheFolder, attachment.UID+".mp4"), nil
|
||||
}
|
||||
|
||||
// =============================================================================
|
||||
// Authentication & Authorization
|
||||
// =============================================================================
|
||||
|
||||
// checkAttachmentPermission verifies the user has permission to access the attachment.
|
||||
func (s *FileServerService) checkAttachmentPermission(ctx context.Context, c *echo.Context, attachment *store.Attachment) error {
|
||||
// For unlinked attachments, only the creator can access.
|
||||
if attachment.MemoID == nil {
|
||||
user, err := s.getCurrentUser(ctx, c)
|
||||
if err != nil {
|
||||
return echo.NewHTTPError(http.StatusInternalServerError, "failed to get current user").Wrap(err)
|
||||
}
|
||||
if user == nil {
|
||||
return echo.NewHTTPError(http.StatusUnauthorized, "unauthorized access")
|
||||
}
|
||||
if user.ID != attachment.CreatorID && user.Role != store.RoleAdmin {
|
||||
return echo.NewHTTPError(http.StatusForbidden, "forbidden access")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
memo, err := s.Store.GetMemo(ctx, &store.FindMemo{ID: attachment.MemoID})
|
||||
if err != nil {
|
||||
return echo.NewHTTPError(http.StatusInternalServerError, "failed to find memo").Wrap(err)
|
||||
}
|
||||
if memo == nil {
|
||||
return echo.NewHTTPError(http.StatusNotFound, "memo not found")
|
||||
}
|
||||
|
||||
if memo.Visibility == store.Public {
|
||||
return nil
|
||||
}
|
||||
|
||||
// Check share token fallback: allow access if request carries a valid, non-expired share token
|
||||
// that was issued for this specific memo. This covers attachment requests made from the shared
|
||||
// memo page for private or protected memos.
|
||||
if shareToken := (*c).QueryParam("share_token"); shareToken != "" {
|
||||
ms, err := s.Store.GetMemoShare(ctx, &store.FindMemoShare{UID: &shareToken})
|
||||
if err == nil && ms != nil && !isMemoShareExpired(ms) && ms.MemoID == memo.ID {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
user, err := s.getCurrentUser(ctx, c)
|
||||
if err != nil {
|
||||
return echo.NewHTTPError(http.StatusInternalServerError, "failed to get current user").Wrap(err)
|
||||
}
|
||||
if user == nil {
|
||||
return echo.NewHTTPError(http.StatusUnauthorized, "unauthorized access")
|
||||
}
|
||||
|
||||
if memo.Visibility == store.Private && user.ID != memo.CreatorID && user.Role != store.RoleAdmin {
|
||||
return echo.NewHTTPError(http.StatusForbidden, "forbidden access")
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// getCurrentUser retrieves the current authenticated user from the request.
|
||||
// Authentication priority: Bearer token (Access Token V2 or PAT) > Refresh token cookie.
|
||||
func (s *FileServerService) getCurrentUser(ctx context.Context, c *echo.Context) (*store.User, error) {
|
||||
authHeader := c.Request().Header.Get(echo.HeaderAuthorization)
|
||||
cookieHeader := c.Request().Header.Get("Cookie")
|
||||
return s.authenticator.AuthenticateToUser(ctx, authHeader, cookieHeader)
|
||||
}
|
||||
|
||||
// getUserByUsername finds a user by username only.
|
||||
func (s *FileServerService) getUserByUsername(ctx context.Context, identifier string) (*store.User, error) {
|
||||
return s.Store.GetUser(ctx, &store.FindUser{Username: &identifier})
|
||||
}
|
||||
|
||||
// =============================================================================
|
||||
// Helper Functions
|
||||
// =============================================================================
|
||||
|
||||
// sanitizeContentType converts potentially dangerous MIME types to safe alternatives.
|
||||
func (*FileServerService) sanitizeContentType(mimeType string) string {
|
||||
contentType := mimeType
|
||||
if strings.HasPrefix(contentType, "text/") {
|
||||
contentType += "; charset=utf-8"
|
||||
}
|
||||
// Normalize for case-insensitive lookup.
|
||||
if xssUnsafeTypes[strings.ToLower(mimeType)] {
|
||||
return "application/octet-stream"
|
||||
}
|
||||
return contentType
|
||||
}
|
||||
|
||||
// parseDataURI extracts MIME type and decoded data from a data URI.
|
||||
func (*FileServerService) parseDataURI(dataURI string) (string, []byte, error) {
|
||||
matches := dataURIRegex.FindStringSubmatch(dataURI)
|
||||
if len(matches) != 3 {
|
||||
return "", nil, errors.New("invalid data URI format")
|
||||
}
|
||||
|
||||
imageType := matches[1]
|
||||
imageData, err := base64.StdEncoding.DecodeString(matches[2])
|
||||
if err != nil {
|
||||
return "", nil, errors.Wrap(err, "failed to decode base64 data")
|
||||
}
|
||||
|
||||
return imageType, imageData, nil
|
||||
}
|
||||
|
||||
// isMediaType checks if the MIME type is video or audio.
|
||||
func isMediaType(mimeType string) bool {
|
||||
return strings.HasPrefix(mimeType, "video/") || strings.HasPrefix(mimeType, "audio/")
|
||||
}
|
||||
|
||||
// setSecurityHeaders sets common security headers for all responses.
|
||||
func setSecurityHeaders(c *echo.Context) {
|
||||
h := c.Response().Header()
|
||||
h.Set("X-Content-Type-Options", "nosniff")
|
||||
h.Set("X-Frame-Options", "DENY")
|
||||
h.Set("Content-Security-Policy", "default-src 'none'; style-src 'unsafe-inline';")
|
||||
}
|
||||
|
||||
// setMediaHeaders sets headers for media file responses.
|
||||
func setMediaHeaders(c *echo.Context, contentType, originalType string) {
|
||||
h := c.Response().Header()
|
||||
h.Set(echo.HeaderContentType, contentType)
|
||||
h.Set(echo.HeaderCacheControl, cacheMaxAge)
|
||||
|
||||
// Support HDR/wide color gamut for images and videos.
|
||||
if strings.HasPrefix(originalType, "image/") || strings.HasPrefix(originalType, "video/") {
|
||||
h.Set("Color-Gamut", "srgb, p3, rec2020")
|
||||
}
|
||||
}
|
||||
|
||||
// isMemoShareExpired returns true if the share has a defined expiry that has already passed.
|
||||
func isMemoShareExpired(ms *store.MemoShare) bool {
|
||||
return ms.ExpiresTs != nil && time.Now().Unix() > *ms.ExpiresTs
|
||||
}
|
||||
@@ -0,0 +1,435 @@
|
||||
package fileserver
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/binary"
|
||||
"fmt"
|
||||
"hash/crc32"
|
||||
"image"
|
||||
"image/color"
|
||||
"image/png"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/labstack/echo/v5"
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"github.com/usememos/memos/internal/markdown"
|
||||
"github.com/usememos/memos/internal/profile"
|
||||
"github.com/usememos/memos/internal/testutil"
|
||||
apiv1 "github.com/usememos/memos/proto/gen/api/v1"
|
||||
"github.com/usememos/memos/server/auth"
|
||||
apiv1service "github.com/usememos/memos/server/router/api/v1"
|
||||
"github.com/usememos/memos/store"
|
||||
teststore "github.com/usememos/memos/store/test"
|
||||
)
|
||||
|
||||
func TestServeAttachmentFile_ShareTokenAllowsDirectMemoAttachment(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
svc, fs, _, cleanup := newShareAttachmentTestServices(ctx, t)
|
||||
defer cleanup()
|
||||
|
||||
creator, err := svc.Store.CreateUser(ctx, &store.User{
|
||||
Username: "share-parent-owner",
|
||||
Role: store.RoleUser,
|
||||
Email: "share-parent-owner@example.com",
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
creatorCtx := context.WithValue(ctx, auth.UserIDContextKey, creator.ID)
|
||||
|
||||
attachment, err := svc.CreateAttachment(creatorCtx, &apiv1.CreateAttachmentRequest{
|
||||
Attachment: &apiv1.Attachment{
|
||||
Filename: "memo.txt",
|
||||
Type: "text/plain",
|
||||
Content: []byte("memo attachment"),
|
||||
},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
parentMemo, err := svc.CreateMemo(creatorCtx, &apiv1.CreateMemoRequest{
|
||||
Memo: &apiv1.Memo{
|
||||
Content: "shared parent",
|
||||
Visibility: apiv1.Visibility_PROTECTED,
|
||||
Attachments: []*apiv1.Attachment{
|
||||
{Name: attachment.Name},
|
||||
},
|
||||
},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
share, err := svc.CreateMemoShare(creatorCtx, &apiv1.CreateMemoShareRequest{
|
||||
Parent: parentMemo.Name,
|
||||
MemoShare: &apiv1.MemoShare{},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
shareToken := share.Name[strings.LastIndex(share.Name, "/")+1:]
|
||||
|
||||
e := echo.New()
|
||||
fs.RegisterRoutes(e)
|
||||
|
||||
req := httptest.NewRequest(http.MethodGet, fmt.Sprintf("/file/%s/%s?share_token=%s", attachment.Name, attachment.Filename, shareToken), nil)
|
||||
rec := httptest.NewRecorder()
|
||||
e.ServeHTTP(rec, req)
|
||||
|
||||
require.Equal(t, http.StatusOK, rec.Code)
|
||||
require.Equal(t, "memo attachment", rec.Body.String())
|
||||
}
|
||||
|
||||
func TestServeAttachmentFile_LocalStaticFileSupportsRangeRequests(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
svc, fs, _, cleanup := newShareAttachmentTestServices(ctx, t)
|
||||
defer cleanup()
|
||||
|
||||
creator, err := svc.Store.CreateUser(ctx, &store.User{
|
||||
Username: "range-owner",
|
||||
Role: store.RoleUser,
|
||||
Email: "range-owner@example.com",
|
||||
})
|
||||
require.NoError(t, err)
|
||||
creatorCtx := context.WithValue(ctx, auth.UserIDContextKey, creator.ID)
|
||||
|
||||
attachment, err := svc.CreateAttachment(creatorCtx, &apiv1.CreateAttachmentRequest{
|
||||
Attachment: &apiv1.Attachment{
|
||||
Filename: "range.txt",
|
||||
Type: "text/plain",
|
||||
Content: []byte("0123456789"),
|
||||
},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
_, err = svc.CreateMemo(creatorCtx, &apiv1.CreateMemoRequest{
|
||||
Memo: &apiv1.Memo{
|
||||
Content: "range memo",
|
||||
Visibility: apiv1.Visibility_PUBLIC,
|
||||
Attachments: []*apiv1.Attachment{
|
||||
{Name: attachment.Name},
|
||||
},
|
||||
},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
e := echo.New()
|
||||
fs.RegisterRoutes(e)
|
||||
|
||||
req := httptest.NewRequest(http.MethodGet, fmt.Sprintf("/file/%s/%s", attachment.Name, attachment.Filename), nil)
|
||||
req.Header.Set("Range", "bytes=2-5")
|
||||
rec := httptest.NewRecorder()
|
||||
e.ServeHTTP(rec, req)
|
||||
|
||||
require.Equal(t, http.StatusPartialContent, rec.Code)
|
||||
require.Equal(t, "2345", rec.Body.String())
|
||||
require.Equal(t, "bytes 2-5/10", rec.Header().Get("Content-Range"))
|
||||
}
|
||||
|
||||
func TestServeAttachmentFile_ShareTokenRejectsCommentAttachment(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
svc, fs, _, cleanup := newShareAttachmentTestServices(ctx, t)
|
||||
defer cleanup()
|
||||
|
||||
creator, err := svc.Store.CreateUser(ctx, &store.User{
|
||||
Username: "private-parent-owner",
|
||||
Role: store.RoleUser,
|
||||
Email: "private-parent-owner@example.com",
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
creatorCtx := context.WithValue(ctx, auth.UserIDContextKey, creator.ID)
|
||||
commenter, err := svc.Store.CreateUser(ctx, &store.User{
|
||||
Username: "share-commenter",
|
||||
Role: store.RoleUser,
|
||||
Email: "share-commenter@example.com",
|
||||
})
|
||||
require.NoError(t, err)
|
||||
commenterCtx := context.WithValue(ctx, auth.UserIDContextKey, commenter.ID)
|
||||
|
||||
parentMemo, err := svc.CreateMemo(creatorCtx, &apiv1.CreateMemoRequest{
|
||||
Memo: &apiv1.Memo{
|
||||
Content: "shared parent",
|
||||
Visibility: apiv1.Visibility_PROTECTED,
|
||||
},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
commentAttachment, err := svc.CreateAttachment(commenterCtx, &apiv1.CreateAttachmentRequest{
|
||||
Attachment: &apiv1.Attachment{
|
||||
Filename: "comment.txt",
|
||||
Type: "text/plain",
|
||||
Content: []byte("comment attachment"),
|
||||
},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
_, err = svc.CreateMemoComment(commenterCtx, &apiv1.CreateMemoCommentRequest{
|
||||
Name: parentMemo.Name,
|
||||
Comment: &apiv1.Memo{
|
||||
Content: "comment with attachment",
|
||||
Visibility: apiv1.Visibility_PROTECTED,
|
||||
Attachments: []*apiv1.Attachment{
|
||||
{Name: commentAttachment.Name},
|
||||
},
|
||||
},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
share, err := svc.CreateMemoShare(creatorCtx, &apiv1.CreateMemoShareRequest{
|
||||
Parent: parentMemo.Name,
|
||||
MemoShare: &apiv1.MemoShare{},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
shareToken := share.Name[strings.LastIndex(share.Name, "/")+1:]
|
||||
|
||||
e := echo.New()
|
||||
fs.RegisterRoutes(e)
|
||||
|
||||
req := httptest.NewRequest(http.MethodGet, fmt.Sprintf("/file/%s/%s?share_token=%s", commentAttachment.Name, commentAttachment.Filename, shareToken), nil)
|
||||
rec := httptest.NewRecorder()
|
||||
e.ServeHTTP(rec, req)
|
||||
|
||||
require.Equal(t, http.StatusUnauthorized, rec.Code)
|
||||
}
|
||||
|
||||
func TestServeAttachmentFile_MotionClip(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
svc, fs, _, cleanup := newShareAttachmentTestServices(ctx, t)
|
||||
defer cleanup()
|
||||
|
||||
creator, err := svc.Store.CreateUser(ctx, &store.User{
|
||||
Username: "motion-owner",
|
||||
Role: store.RoleUser,
|
||||
Email: "motion-owner@example.com",
|
||||
})
|
||||
require.NoError(t, err)
|
||||
creatorCtx := context.WithValue(ctx, auth.UserIDContextKey, creator.ID)
|
||||
|
||||
attachment, err := svc.CreateAttachment(creatorCtx, &apiv1.CreateAttachmentRequest{
|
||||
Attachment: &apiv1.Attachment{
|
||||
Filename: "motion.jpg",
|
||||
Type: "image/jpeg",
|
||||
Content: testutil.BuildMotionPhotoJPEG(),
|
||||
},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
_, err = svc.CreateMemo(creatorCtx, &apiv1.CreateMemoRequest{
|
||||
Memo: &apiv1.Memo{
|
||||
Content: "motion memo",
|
||||
Visibility: apiv1.Visibility_PUBLIC,
|
||||
Attachments: []*apiv1.Attachment{
|
||||
{Name: attachment.Name},
|
||||
},
|
||||
},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
e := echo.New()
|
||||
fs.RegisterRoutes(e)
|
||||
|
||||
req := httptest.NewRequest(http.MethodGet, fmt.Sprintf("/file/%s/%s?motion=true", attachment.Name, attachment.Filename), nil)
|
||||
rec := httptest.NewRecorder()
|
||||
e.ServeHTTP(rec, req)
|
||||
|
||||
require.Equal(t, http.StatusOK, rec.Code)
|
||||
require.Equal(t, "video/mp4", rec.Header().Get("Content-Type"))
|
||||
require.Contains(t, rec.Body.String(), "ftyp")
|
||||
}
|
||||
|
||||
func TestServeAttachmentFile_SVGThumbnailServedAsImageWithSecurityHeaders(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
svc, fs, _, cleanup := newShareAttachmentTestServices(ctx, t)
|
||||
defer cleanup()
|
||||
|
||||
creator, err := svc.Store.CreateUser(ctx, &store.User{
|
||||
Username: "svg-owner",
|
||||
Role: store.RoleUser,
|
||||
Email: "svg-owner@example.com",
|
||||
})
|
||||
require.NoError(t, err)
|
||||
creatorCtx := context.WithValue(ctx, auth.UserIDContextKey, creator.ID)
|
||||
|
||||
svgContent := []byte(`<svg xmlns="http://www.w3.org/2000/svg" width="120" height="40"><text x="0" y="20">memos</text></svg>`)
|
||||
attachment, err := svc.CreateAttachment(creatorCtx, &apiv1.CreateAttachmentRequest{
|
||||
Attachment: &apiv1.Attachment{
|
||||
Filename: "preview.svg",
|
||||
Type: "image/svg+xml",
|
||||
Content: svgContent,
|
||||
},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
_, err = svc.CreateMemo(creatorCtx, &apiv1.CreateMemoRequest{
|
||||
Memo: &apiv1.Memo{
|
||||
Content: "svg memo",
|
||||
Visibility: apiv1.Visibility_PUBLIC,
|
||||
Attachments: []*apiv1.Attachment{
|
||||
{Name: attachment.Name},
|
||||
},
|
||||
},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
e := echo.New()
|
||||
fs.RegisterRoutes(e)
|
||||
|
||||
req := httptest.NewRequest(http.MethodGet, fmt.Sprintf("/file/%s/%s?thumbnail=true", attachment.Name, attachment.Filename), nil)
|
||||
rec := httptest.NewRecorder()
|
||||
e.ServeHTTP(rec, req)
|
||||
|
||||
require.Equal(t, http.StatusOK, rec.Code)
|
||||
require.Equal(t, "image/svg+xml", rec.Header().Get("Content-Type"))
|
||||
require.Empty(t, rec.Header().Get("Content-Disposition"))
|
||||
require.Equal(t, "nosniff", rec.Header().Get("X-Content-Type-Options"))
|
||||
require.Equal(t, "default-src 'none'; style-src 'unsafe-inline';", rec.Header().Get("Content-Security-Policy"))
|
||||
require.Equal(t, svgContent, rec.Body.Bytes())
|
||||
}
|
||||
|
||||
func TestServeAttachmentFile_ThumbnailWithSensitiveMetadataServesOriginal(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
svc, fs, _, cleanup := newShareAttachmentTestServices(ctx, t)
|
||||
defer cleanup()
|
||||
|
||||
creator, err := svc.Store.CreateUser(ctx, &store.User{
|
||||
Username: "hdr-owner",
|
||||
Role: store.RoleUser,
|
||||
Email: "hdr-owner@example.com",
|
||||
})
|
||||
require.NoError(t, err)
|
||||
creatorCtx := context.WithValue(ctx, auth.UserIDContextKey, creator.ID)
|
||||
|
||||
imageContent := testPNGWithChunk(t, "cICP", []byte{9, 16, 9, 1})
|
||||
attachment, err := svc.CreateAttachment(creatorCtx, &apiv1.CreateAttachmentRequest{
|
||||
Attachment: &apiv1.Attachment{
|
||||
Filename: "hdr.png",
|
||||
Type: "image/png",
|
||||
Content: imageContent,
|
||||
},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
_, err = svc.CreateMemo(creatorCtx, &apiv1.CreateMemoRequest{
|
||||
Memo: &apiv1.Memo{
|
||||
Content: "hdr memo",
|
||||
Visibility: apiv1.Visibility_PUBLIC,
|
||||
Attachments: []*apiv1.Attachment{
|
||||
{Name: attachment.Name},
|
||||
},
|
||||
},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
e := echo.New()
|
||||
fs.RegisterRoutes(e)
|
||||
|
||||
req := httptest.NewRequest(http.MethodGet, fmt.Sprintf("/file/%s/%s?thumbnail=true", attachment.Name, attachment.Filename), nil)
|
||||
rec := httptest.NewRecorder()
|
||||
e.ServeHTTP(rec, req)
|
||||
|
||||
require.Equal(t, http.StatusOK, rec.Code)
|
||||
require.Equal(t, "image/png", rec.Header().Get("Content-Type"))
|
||||
require.Equal(t, imageContent, rec.Body.Bytes())
|
||||
}
|
||||
|
||||
func TestHasThumbnailSensitiveMetadata(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
mimeType string
|
||||
data []byte
|
||||
want bool
|
||||
}{
|
||||
{
|
||||
name: "jpeg hdr gain map",
|
||||
mimeType: "image/jpeg",
|
||||
data: []byte("xmp hdrgm:Version=\"1.0\""),
|
||||
want: true,
|
||||
},
|
||||
{
|
||||
name: "jpeg icc profile",
|
||||
mimeType: "image/jpeg",
|
||||
data: []byte("ICC_PROFILE"),
|
||||
want: true,
|
||||
},
|
||||
{
|
||||
name: "png cicp chunk",
|
||||
mimeType: "image/png",
|
||||
data: []byte("cICP"),
|
||||
want: true,
|
||||
},
|
||||
{
|
||||
name: "heic",
|
||||
mimeType: "image/heic",
|
||||
data: nil,
|
||||
want: true,
|
||||
},
|
||||
{
|
||||
name: "plain jpeg",
|
||||
mimeType: "image/jpeg",
|
||||
data: []byte("plain image data"),
|
||||
want: false,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
require.Equal(t, tt.want, hasThumbnailSensitiveMetadata(tt.mimeType, tt.data))
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func testPNGWithChunk(t *testing.T, chunkType string, chunkData []byte) []byte {
|
||||
t.Helper()
|
||||
|
||||
img := image.NewRGBA(image.Rect(0, 0, 1, 1))
|
||||
img.Set(0, 0, color.RGBA{R: 255, A: 255})
|
||||
|
||||
var encoded bytes.Buffer
|
||||
require.NoError(t, png.Encode(&encoded, img))
|
||||
|
||||
pngData := encoded.Bytes()
|
||||
iendIndex := bytes.LastIndex(pngData, []byte("IEND"))
|
||||
require.GreaterOrEqual(t, iendIndex, 4)
|
||||
|
||||
chunkStart := iendIndex - 4
|
||||
var chunk bytes.Buffer
|
||||
require.NoError(t, binary.Write(&chunk, binary.BigEndian, uint32(len(chunkData))))
|
||||
chunk.WriteString(chunkType)
|
||||
chunk.Write(chunkData)
|
||||
checksum := crc32.ChecksumIEEE(append([]byte(chunkType), chunkData...))
|
||||
require.NoError(t, binary.Write(&chunk, binary.BigEndian, checksum))
|
||||
|
||||
result := make([]byte, 0, len(pngData)+chunk.Len())
|
||||
result = append(result, pngData[:chunkStart]...)
|
||||
result = append(result, chunk.Bytes()...)
|
||||
result = append(result, pngData[chunkStart:]...)
|
||||
return result
|
||||
}
|
||||
|
||||
func newShareAttachmentTestServices(ctx context.Context, t *testing.T) (*apiv1service.APIV1Service, *FileServerService, *store.Store, func()) {
|
||||
t.Helper()
|
||||
|
||||
testStore := teststore.NewTestingStore(ctx, t)
|
||||
testProfile := &profile.Profile{
|
||||
Demo: true,
|
||||
Version: "test-1.0.0",
|
||||
InstanceURL: "http://localhost:8080",
|
||||
Driver: "sqlite",
|
||||
DSN: ":memory:",
|
||||
Data: t.TempDir(),
|
||||
}
|
||||
secret := "test-secret"
|
||||
markdownService := markdown.NewService(markdown.WithTagExtension())
|
||||
apiService := &apiv1service.APIV1Service{
|
||||
Secret: secret,
|
||||
Profile: testProfile,
|
||||
Store: testStore,
|
||||
MarkdownService: markdownService,
|
||||
SSEHub: apiv1service.NewSSEHub(),
|
||||
}
|
||||
fileService := NewFileServerService(testProfile, testStore, secret)
|
||||
|
||||
return apiService, fileService, testStore, func() {
|
||||
testStore.Close()
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user